Communication system, information processing system, and information processing method

ABSTRACT

A communication system for controlling communication between a terminal apparatus and a counterpart terminal apparatus, includes circuitry that: converts speech data into language information, the speech data representing speech input to the terminal apparatus or speech input to the counterpart terminal apparatus; displays the language information that is converted on an information display screen at the terminal apparatus in a chronological order; displays, on the information display screen, content based on a user operation on the language information displayed in the first area of the information display screen; and transmits information on the content displayed on the information display screen to a counterpart terminal apparatus, to cause the counterpart terminal apparatus to display the content.

CROSS-REFERENCE TO RELATED APPLICATIONS

This patent application is based on and claims priority pursuant to 35U.S.C. § 119(a) to Japanese Patent Application Nos. 2022-082411, filedon May 19, 2022, and 2023-047032, filed on Mar. 23, 2023, in the JapanPatent Office, the entire disclosure of which is hereby incorporated byreference herein.

BACKGROUND Technical Field

The present invention relates to a communication system, an informationprocessing system, and an information processing method.

Related Art

Communication using language information alone may not be sufficient toefficiently convey information especially in a short time. For example,in a meeting to discuss a new plan or idea, use of visual informationwould be more effective in communication.

SUMMARY

Example embodiments include a communication system for controllingcommunication between a terminal apparatus and a counterpart terminalapparatus, which includes circuitry that: converts speech data intolanguage information, the speech data representing speech input to theterminal apparatus or speech input to the counterpart terminalapparatus; displays the language information that is converted on aninformation display screen at the terminal apparatus in a chronologicalorder; displays, on the information display screen, content based on auser operation on the language information displayed on the informationdisplay screen; and transmits information on the content displayed onthe information display screen to a counterpart terminal apparatus, tocause the counterpart terminal apparatus to display the content.

Example embodiments include an information processing system includingcircuitry that: converts speech data into language information, thespeech data representing speech input to a terminal apparatus or speechinput to a counterpart terminal apparatus, the terminal apparatus andthe counterpart terminal apparatus being configured to communicate via anetwork; displays the language information that is converted on aninformation display screen of the terminal apparatus in a chronologicalorder; and in response to a user operation on the language informationdisplayed on the information display screen, transmits information oncontent displayed based on the user operation to the counterpartterminal apparatus, to cause the counterpart terminal apparatus todisplay the content.

Example embodiments include an information processing method including:converting speech data into language information, the speech datarepresenting speech input to a terminal apparatus or speech input to acounterpart terminal apparatus, the terminal apparatus and thecounterpart terminal apparatus being configured to communicate via anetwork; displaying the language information that is converted on aninformation display screen of the terminal apparatus in a chronologicalorder; and in response to a user operation on the language informationdisplayed on the information display screen, transmitting information oncontent displayed based on the user operation to the counterpartterminal apparatus, to cause the counterpart terminal apparatus todisplay the content.

BRIEF DESCRIPTION OF THE DRAWINGS

A more complete appreciation of embodiments of the present disclosureand many of the attendant advantages and features thereof can be readilyobtained and understood from the following detailed description withreference to the accompanying drawings, wherein:

FIG. 1 is an example of an information display screen displayed by aterminal apparatus during a meeting;

FIG. 2 is a diagram illustrating a configuration of a visualcommunication system, according to an exemplary embodiment;

FIG. 3 is a block diagram illustrating a hardware configuration of acomputer according to an exemplary embodiment;

FIG. 4 is a block diagram illustrating a functional configuration of thevisual communication system according to an exemplary embodiment;

FIG. 5 is a conceptual diagram illustrating an example of conversationtable;

FIG. 6 is a conceptual diagram illustrating an example of content table;

FIG. 7 is a conceptual diagram illustrating an example of illustrationtable;

FIG. 8 is a conceptual diagram illustrating an example of meetingmanagement table;

FIGS. 9A and 9B are a conceptual diagram illustrating an example ofcontent log table;

FIG. 10 is a conceptual diagram illustrating an example of preferredlanguage information table;

FIG. 11 is an illustration of an information display screen in aninitial state, which is displayed by the terminal apparatus, accordingto an exemplary embodiment;

FIG. 12 is a sequence diagram illustrating processing of transmittingspeech data from the terminal apparatus to the information processingsystem, and processing to share language information converted from thespeech data at the information processing system with the counterpartterminal apparatus, according to an exemplary embodiment;

FIGS. 13A to 13D are schematic diagrams illustrating processing ofcopying language information from the conversation log area to thewhiteboard work area, performed by the terminal apparatus, in responseto a user operation, according to an exemplary embodiment;

FIGS. 14A and 14B are diagrams for explaining the display order ofthumbnails of illustrations, which are determined based on languageinformation, according to an exemplary embodiment;

FIG. 15 is a sequence diagram illustrating processing of copyinglanguage information from the conversation log area to the whiteboardwork area, performed by the terminal apparatus, in response to a useroperation, according to an exemplary embodiment;

FIGS. 16A to 16D are schematic diagrams illustrating processing ofcopying full text of language information from the conversation log areato the whiteboard work area, performed by the terminal apparatus, inresponse to a user operation, according to an exemplary embodiment;

FIGS. 17A to 17C are diagrams illustrating a display example of anillustration, searched based on the language information copied to thewhiteboard work area;

FIG. 18 is a sequence diagram illustrating processing of displaying anillustration based on language information of the whiteboard work area,performed by the terminal apparatus, in response to a user operation,according to an exemplary embodiment;

FIGS. 19A to 19C are diagrams for explaining decomposition of languageinformation copied to the whiteboard work area into character strings;

FIG. 20 is a sequence diagram illustrating processing of decomposinglanguage information in the whiteboard work area into character strings,performed by the terminal apparatus, in response to a user operation,according to an exemplary embodiment;

FIGS. 21A to 21C are diagrams for explaining hierarchization of aplurality of contents displayed in the whiteboard work area, accordingto an exemplary embodiment;

FIG. 22 is a block diagram illustrating a functional configuration of anode position output based on machine learning, according to anexemplary embodiment;

FIG. 23 is a sequence diagram illustrating hierarchization of languageinformation in the whiteboard work area, performed by the terminalapparatus, in response to a user operation, according to an exemplaryembodiment;

FIGS. 24A to 24C illustrate the language information in the conversationlog area, which is highlighted when the content in the whiteboard workarea is selected, according to an exemplary embodiment;

FIG. 25 is a sequence diagram illustrating processing of displaying thelanguage information corresponding to the content in the whiteboard workarea, with emphasis, performed by the terminal apparatus, in response toa user operation, according to an exemplary embodiment;

FIG. 26 is a flowchart illustrating processing of filtering languageinformation in the conversation log area, performed by the terminalapparatus, according to an exemplary embodiment.

FIG. 27 is a sequence diagram illustrating processing of translatinglanguage information, and transmitting the translated languageinformation to the counterpart terminal apparatus, performed by theinformation processing system, according to an exemplary embodiment;

FIGS. 28A to 28C are diagrams for explaining processing to review thediscussions in the meeting (meeting contents), when the discussion inthe meeting are reproduced according to user operation, according to anexemplary embodiment;

FIGS. 29A and 29B are diagrams for explaining processing to review thediscussions in the meeting (meeting contents) when the discussions(meeting contents) in the meeting are automatically reproduced,according to an exemplary embodiment;

FIG. 30 is an illustration of an example slider bar; and

FIG. 31 is a flowchart illustrating processing of playing back thewhiteboard work area, performed by the playback unit, when executing thereview mode, according to an exemplary embodiment.

The accompanying drawings are intended to depict embodiments of thepresent disclosure and should not be interpreted to limit the scopethereof. The accompanying drawings are not to be considered as drawn toscale unless explicitly noted. Also, identical or similar referencenumerals designate identical or similar components throughout theseveral views.

DETAILED DESCRIPTION

In describing embodiments illustrated in the drawings, specificterminology is employed for the sake of clarity. However, the disclosureof this specification is not intended to be limited to the specificterminology so selected and it is to be understood that each specificelement includes all technical equivalents that have a similar function,operate in a similar manner, and achieve a similar result.

Referring now to the drawings, embodiments of the present disclosure aredescribed below. As used herein, the singular forms “a,” “an,” and “the”are intended to include the plural forms as well, unless the contextclearly indicates otherwise.

DETAILED DESCRIPTION

Hereinafter, as an example of an embodiment for carrying out the presentinvention, a visual communication system and an information processingmethod performed by the visual communication system will be describedwith reference to the drawings.

<Outline of Operation>

Communication using language information alone may not be sufficient toefficiently convey information in a short time. For example, in ameeting to discuss a new plan or idea, use of visual information wouldbe more effective in communication. Although the user may draw someillustrations to describe idea to visualize, it may not be easy for someusers to illustrate what he or she thinks from the scratch. In view ofthis, a visual communication system is provided, which converts languageinformation into visual information such as an illustration, toefficiently convey the information that the user is trying to describeusing the language information.

Referring to FIG. 1 , an overview of processing performed by the visualcommunication system is described according to one or more exemplaryembodiments. FIG. 1 illustrates an information display screen 100displayed by a terminal apparatus during a meeting. As illustrated inFIG. 1 , the information display screen 100 includes a conversation logarea 101 (an example of a first area) and a whiteboard work area 102 (anexample of a second area). The whiteboard work area 102 is a displayarea of a touch panel, and is an area that allows a meeting participant(“participant”) to display hand-drafted data, language information thatis directly input, or various images, substantially in the same manneras a display area of the known electronic whiteboard. The conversationlog area 101 is an area, which displays a log of language information,having been converted from contents of speech of the participant throughspeech recognition. The whiteboard work area 102 does not have to beimplemented by a touch panel, such that any display capable ofdisplaying, without the touch panel, may be used. For example, the usermay input hand-drafted data using such as a mouse.

As described above, the whiteboard work area 102 and the conversationlog area 101 are displayed on the same screen. The participant is ableto relocate the language information displayed in the conversation logarea 101, in the whiteboard work area 102, by clicking or dragging thelanguage information. In FIG. 1 , a character string 104 “relaxed” and acharacter string 105 “face-to-face communication is better” aredisplayed in the whiteboard work area 102.

In addition, the participant is able to place an illustration in thewhiteboard work area 102 based on the language information in theconversation log area 101 or the whiteboard work area 102. In FIG. 1 ,an illustration 108 based on the character string 105 “face-to-facecommunication is better” is displayed in the whiteboard work area 102.

As described above, the visual communication system 5 according to theembodiment can display visual information such as an illustration in thewhiteboard work area 102 based on language information converted fromspeech. The user can easily visualize the communication during themeeting, as the illustration is automatically displayed. That is, theuser does not have to convert the contents of conversation (contents ofspeech) into an illustration, draw the illustration in the whiteboardwork area 102, or acquire the illustration from a network. Since thewhiteboard work area 102 and the conversation log area 101 are displayedon the same screen, the user is able to instantly know a flow ofcommunication in which each content in the whiteboard work area 102 isproduced.

Terminology

The language information is made up of words in any desired languagesuch as Japanese or English. In this embodiment, the languageinformation includes not only information obtained by speech recognitionof utterance by a speaker but also information input from such as akeyboard. The language information includes information that can bedisplayed by a computer based on character codes, such as characters,figures or shapes, numbers, and alphabets.

Processing of “storing or accumulating language information in achorological order” means that the language information is displayed inthe order of utterance. The language information, which is accumulated,can be viewed any time.

The work area is an area on a computer screen where the user can editarrangement or contents of the language information, or arrangement orcontents of any other content. As an example work area, there is an areathat can be used by the user, such as a display area of an electronicwhiteboard. The electronic whiteboard is a device, which allows a userto write hand-drafted data using a touch panel in a similar manner asthe conventional whiteboard, while displaying a screen or data availablefrom any device such as a PC. The whiteboard may be referred to as anelectronic blackboard, an electronic information board, or an electronicwhiteboard, etc.

The content is a display object displayed in the work area. The contentmay be any object that can be displayed by a computer and visuallyrecognized by a user, such as language information, illustrations,figures, tables, any other data such as screen data that can beprocessed by the computer, or data on the Internet.

Example of System Configuration

FIG. 2 is a diagram illustrating a configuration of the visualcommunication system 5, according to the exemplary embodiment. Thevisual communication system 5 includes a terminal apparatus 1 and aninformation processing system 3 communicably connected with each othervia a network 2.

As described in the following, a number of terminal apparatus 1 is notlimited to one, such that any number of terminal apparatus may beconnected to the network 2 to establish communication. In the following,it is assumed that the terminal apparatus 1 communicates with anotherterminal apparatus, which is referred to as a counterpart terminalapparatus 9. While the counterpart terminal apparatus 9 is substantiallysimilar to the terminal apparatus 1 in operation, it is referred to asthe counterpart terminal apparatus 9 for the descriptive purpose.

The network 2 may be implemented by any one or more of the Internet, amobile communication network, a local area network (LAN), for example.The network 2 may include a wired communication network and a wirelesscommunication network. The wireless communication network may be basedon a wireless communication standard such as third generation (3G),fourth generation (4G), fifth generation (5G), Wireless Fidelity(Wi-Fi®), Worldwide Interoperability for Microwave Access (WiMAX), orLong Term Evolution (LTE).

The terminal apparatus 1 is an information processing apparatus operatedby a user such as the participant. The terminal apparatus 1 transmitsspeech data representing utterances of the participant to theinformation processing system 3. The terminal apparatus 1 receives thelanguage information, which is obtained by converting the speech datainto text data at the server, and displays the language information inthe conversation log area. The terminal apparatus 1 repeats thisprocessing, for example, during the meeting. Specifically, the terminalapparatuses respectively operated by a plurality of participants whoparticipate in the meeting (including the terminal apparatus 1 and thecounterpart terminal apparatus 9) are connected to the network 2. Theterminal apparatuses each performs the above-described processing, sothat the language information input by each participant is displayed inthe conversation log area 101 in a chronological order.

The terminal apparatus 1 may be, for example, a desktop personalcomputer (PC), a notebook PC, a smartphone, a tablet terminal, or thelike, which operates on the general OS such as Windows®, MAC® OS,Android®, or iOS®. In addition, the terminal apparatus 1 may be anyapparatus on which an application such as a web browser can operate.

The terminal apparatus 1 may be an electronic whiteboard, an outputdevice such as digital signage, a head-up display (HUD) device, anindustrial machine, a medical device, a network home appliance, aconnected car, a mobile phone (such as a WAP or 3G-compliant phone), asmartphone, a tablet terminal, a game machine, a personal digitalassistant (PDA), or the like.

The information processing system 3 is, for example, an example ofinformation processing apparatus having a function of providinginformation or a processing result in response to a request from theterminal apparatus 1. The information processing system 3 may be calledan information processing server, or simply a server, which may beimplemented by one or more information processing apparatuses. In thevisual communication system 5 of the embodiment, the informationprocessing system 3 may provide an image such as an illustrationcorresponding to language information to the terminal apparatus 1 inaddition to performing the above-described speech recognition. Thespeech recognition may alternatively be performed by the terminalapparatus 1.

The information processing system 3 preferably has a function of a Webserver. The Web server transmits various kinds of information such as anHTML document existing on the Web server, which is specified by a URLinstructed from a Web browser as a client, according to HTTP used forconnection with the terminal apparatus 1. Web application is executed,based on communication between the Web browser and the Web server.Specifically, the Web application is an application that is executed, asa program on the Web browser and a program on the Web server operate incooperation. The program on the Web browser may be a program written ina programming language for Web browser such as JavaScript®. In thisdisclosure, an application, which is not executed unless installed inthe terminal apparatus 1, is referred to as a native application.

The information processing system 3 may support cloud computing. The“cloud computing” refers to Internet-based computing where resources ona network are used or accessed without identifying specific hardwareresources. For example, the web server function, the speech recognitionfunction, and the illustration search function, of the informationprocessing system 3, may be separately implemented by a plurality ofservers.

Example of Hardware Configuration FIG. 3 is a diagram illustrating ahardware configuration of the terminal apparatus 1 and the informationprocessing system 3. The terminal apparatus 1 and the informationprocessing system 3 each have a hardware configuration of a computer 500as illustrated in FIG. 3 .

As illustrated in FIG. 3 , the computer 500 includes a centralprocessing unit (CPU) 501, a read only memory (ROM) 502, a random accessmemory (RAM) 503, a hard disk (HD) 504, a hard disk drive (HDD)controller 505, a display 506, an external device connection interface(I/F) 508, a network I/F 509, a bus line 510, a keyboard 511, a pointingdevice 512, an optical drive 514, a medium I/F 516, a motion capturingdevice 517, a microphone 518, a speaker 519, and a camera 520.

The CPU 501 controls entire operation of the computer 500. The CPU 501may be implemented by any desired or any number of processor. The ROM502 stores a program for controlling the CPU 501, such as an initialprogram loader (IPL). The RAM 503 is used as a work area for the CPU501. The RAM 503 may be implemented by any desired volatile ornon-volatile memory. The HD 504 stores various data such as a program.The HD 504, which may be referred as a hard disk drive, may beimplemented by any desired non-volatile memory capable of storing alarge amount of data. The HD 504 may be provided externally as long asthe HD 504 is accessible. The HDD controller 505 controls reading andwriting of various data from and to the HD 504 under control of the CPU501. The display 506, as an example of displaying means, displaysvarious information such as a cursor, menu, window, character, or image.The external device connection I/F 508 is an interface that connects thecomputer 500 to various extraneous sources. Examples of the externaldevice include, but not limited to, a universal serial bus (USB) memoryand a printer. The network I/F 509 is an interface for communicatingdata via the network 2. The bus line 510 is an address bus, a data bus,or the like for electrically connecting the components such as the CPU501 illustrated in FIG. 3 .

The keyboard 511 is an example of an input device provided with aplurality of keys for allowing a user to enter characters, numericalvalues, or various instructions. The pointing device 512 is an exampleof an input device that allows a user to select or execute a specificinstruction, select a target for processing, or move a cursor beingdisplayed. The optical drive 514 controls reading or writing of variouskinds of data from or to an optical recording medium 513 that is anexample of a removable recording medium. The optical drive 514 may be acompact disc (CD), a digital versatile disc (DVD), a Blu-RayR, or thelike. The medium I/F 516 controls reading and writing (storing) of datafrom and to a recording medium 515 such as a flash memory.

The motion capturing device 517, which is an example of input means, isa circuit that detects a user's motion and converts the detected motioninto an electrical signal. The motion of the user may be detected by anyone of an optical system that detects reflection of light, a magneticsensor attached to the user body, a mechanical sensor, a system thatuses a detection result of a magnetic sensor, a system that analyzes animage of the user, and the like, or a combination thereof.

The microphone 518 is a built-in circuit that converts sound into anelectric signal. The speaker 519 is a built-in circuit that generatessound such as music or voice by converting an electric signal intophysical vibration. The microphone 518 and the speaker 519 may bebuilt-in or provided separately.

The speaker 519 is a built-in circuit that generates sound such as musicor voice by converting an electric signal into physical vibration.

The camera 520 is an example of built-in imaging device capable ofcapturing a target to obtain captured image data.

In case of the computer 500 operating as the information processingsystem 3, the HDD controller 505 controls the HD 504 to store theprogram. In case of the computer 500 operating as the terminal apparatus1, the program is downloaded from the information processing system 3via the network 2 and stored in the HD 504. The CPU 501 then reads theprogram from the HD 504 to execute so as to deploy the functionalconfiguration of FIG. 4 in the RAM 503, collectively at the time ofcompilation or sequentially in accordance with the progress ofprocessing. Further, in case of the computer 500 operating as theinformation processing system 3, the CPU 501 executes the program so asto deploy the functional configuration of FIG. 4 in the RAM 503,collectively at the time of compilation or in accordance with theprogress of processing.

For the computer 500 operating as the information processing system 3,the configuration related to the user interface may be omitted. Forexample, at least one of the display 506, the keyboard 511, the pointingdevice 512, and the motion capturing device 517 may be omitted.

Functional Configuration of Visual Communication System

Next, referring to FIGS. 4 to 10 , a functional configuration of thevisual communication system is described according to embodiments. FIG.4 is a block diagram illustrating a functional configuration of thevisual communication system 5, according to the exemplary embodiment.

Functional Configuration of Information Processing System

As illustrated in FIG. 4 , the information processing system 3 includesa second communication unit 31, a converter 32, a conversation manager33, a content manager 34, and a translation unit 35. These units of theinformation processing system 3 are functions or means implemented by orcaused to function by operating one or more hardware componentsillustrated in FIG. 3 in cooperation with instructions of the CPU 501according to the program loaded from the HD 504 to the RAM 503. Theinformation processing system 3 also includes a storage unit 3000implemented by the RAM 503 and HD 504 illustrated in FIG. 3 .

The second communication unit 31 communicates with the terminalapparatus 1 via the network 2. The second communication unit 31 receivesspeech data or hand-drafted data from the terminal apparatus 1, andtransmits language information converted from the speech data orhand-drafted data to the terminal apparatus 1.

The converter 32 performs recognition processing on speech datarepresenting utterances by the user, or one or more pieces of strokedata (example of hand-drafted data) drafted by the user, to convert suchspeech data or hand-drafted data into language information.

The substance of the language information is a character code. Suchrecognition includes character recognition. Specifically, the converter32 recognizes characters (multiple languages such as not only Japanesebut also English), numbers, symbols (%, $, &, and the like), figures orshapes (line, circle, triangle, and the like), etc., concurrently withthe user's utterance or the user's input operation (such as using apen). Although various algorithms have been proposed for the recognitionmethod, a detailed description is omitted, as any desired knowntechnique can be used in the present embodiment. Further, more than onealgorithm may be used, such that there may be the algorithm suitable torecognition of speech data, and the algorithm suitable to recognition ofhand-drafted data. In this example, the converter 32 performsrecognition processing on speech data indicating speech informationbased on utterance made by the user, and/or performs recognitionprocessing on one or more pieces of stroke data (example of hand-drafteddata) drafted by the user, to convert such speech data or hand-drafteddata into language information. In the present embodiment, the user'sutterance is referred to as “speech” and data obtained by converting thespeech is referred to as “speech data” for distinction. They are notstrictly distinguished from each other for the description purposeunless such distinction is necessary.

The conversion by the converter 32 may be alternatively performed at theterminal apparatus 1. The converter 32 may be provided in another serverindependent of the information processing system 3.

The conversation manager 33 stores the language information, which isconverted from the speech data obtained from each terminal apparatus 1by the converter 32, in the conversation storage unit 3001. Theconversation storage unit 3001 stores a log of conversations in achorological order.

The content manager 34 manages content, that is, language information,illustrations, and the like, displayed in the whiteboard work area 102.The content is stored in a content storage unit 3002 described later.Since the whiteboard work area 102 is shared between the terminalapparatus 1 and the counterpart terminal apparatus 9, the contentstorage unit 3002 includes a plurality of contents to be placed(arranged), or have been placed (arranged), by a user of each terminalapparatus 1.

When the language used by the terminal apparatus 1 is different from thelanguage used by the counterpart terminal apparatus 9, the translationunit 35 translates the language information from the language used bythe terminal apparatus 1 to the language used by the counterpartterminal apparatus 9. Similarly, the translation unit 35 translates thelanguage information from the language used by the counterpart terminalapparatus 9 to the language used by the terminal apparatus 1.

Conversation Table

FIG. 5 is a conceptual diagram illustrating an example of conversationtable. The storage unit 3000 includes the conversation storage unit3001, implemented by a conversation table as illustrated in FIG. 5 . Theconversation table stores a plurality of items including languageinformation ID (“language ID”), language information, input time,speaker, and number of likes, in association.

The language information ID item is an identifier of the languageinformation displayed in the conversation log area 101. The conversationmanager 33 assigns a unique number to each language information, as theidentifier of the language information.

The language information item is language information displayed in theconversation log area 101, which is converted from speech data by theconverter 32.

The input time item is a time when the language information is input,and is, for example, a time when the converter 32 completes convertingfrom the speech data to the language information.

The speaker item is an identifier of a user who has uttered the languageinformation as speech data. In the following, the identifier of the usermay be referred to as a user ID. The identifier of the user istransmitted to the information processing system 3 together with thespeech data.

The number of likes item is a number of times the “like” button ispressed for a particular language information. The “like” button means afavorable response, and the number of likes is the number of favorableresponses.

Content Table

FIG. 6 is a conceptual diagram illustrating an example of content table.The storage unit 3000 includes the content storage unit 3002,implemented by a content table as illustrated in FIG. 5 . The contenttable stores a plurality of items including a content ID, a type, apage, coordinates, a size, a font size, an arrangement time, andlanguage information ID (“language ID”), in association with oneanother.

The content ID item is identification information for identifying acontent. The content manager 34 assigns a content ID to each content.

The type item indicates a type of content. Examples of type includeshand-drafted data, language information, a shape (square, triangle, orthe like), an illustration, a table, and the like. “Hand drafted”indicates stroke data (coordinate point sequence). The languageinformation indicates one or more characters, symbols, etc. (referred toas character codes) arranged in the whiteboard work area 102. Shapeindicates a geometric shape, such as a triangle and a quadrangle, eachof which is converted from the hand-drafted data or the languageinformation. The illustration is image data in any desired data formatsuch as JPEG, Png, or Tiff, which is retrieved using languageinformation or directly placed by the user. The table is an example ofcontent in one-dimensional table form or two-dimensional table form.

In this disclosure, one screen of the whiteboard work area 102 isreferred to as one page. The page item indicates a page number of suchpage.

The coordinates item indicates a position of the content with respect toa predetermined origin of the whiteboard work area 102. The position ofthe content corresponds to, for example, the upper left vertex of thecircumscribed rectangle of the content. The coordinates are expressed,for example, in units of pixel of the display.

The size item indicates a width and a height of the circumscribedrectangle of the content.

The font size item indicates a size of one character, which is a contentwhose type is language information. For the language information, inaddition to the character size, any other property such as a color or afont may be set.

The arrangement time item is a date and time when the content is firstplaced in the whiteboard work area 102. The content may be placed bycopying a content from the conversation log area 101, or by a userdirectly hand-drafting on the whiteboard work area 102. The arrangementtime keeps the time when the content is firstly input, such that thearrangement time is not changed even when the language information isdecomposed by the morphological analysis or the size of the illustrationis changed. This is to maintain the relationship of the arrangement timewith the input time of the language information in the conversation logarea 101.

The language information ID item is the same as the language informationID of specific language information, when the language information orthe illustration is derived from the specific language information inthe conversation log area 101 (for example, when the illustration iscopied or searched using the language information in the conversationlog area 101). Through the language information ID in the content table,each content (in this example, language information or illustration) isassociated with the language information in the conversation log area101. The language information ID is not assigned to the content in theconversation log area 101, which is not derived from the languageinformation in the conversation log area 101.

Illustration Table

FIG. 7 is a conceptual diagram illustrating an example of illustrationtable. The storage unit 3000 includes an illustration storage unit 3003,implemented by an illustration table as illustrated in FIG. 7 . Theillustration table stores a plurality of items including a keyword, anID, and a file name in association with one another.

The keyword item is a character string to be searched for, whichcorresponds to a name or appellation of an illustration to be searched.In this example, the illustration having a file name, which isassociated with a keyword at least partially matching the words(characters) included in the language information, is displayed at theterminal apparatus 1. The keyword may be represented by a word,phonogram, ideogram, sentence, number, or any combination thereof.

The ID item is identification information identifying the illustration.

The file name item is a name of an electronic file of the illustration.

The illustration table as illustrated in FIG. 7 is registered in advancein the information processing system 3 by a system administrator whomanages the information processing system 3, for example. Alternatively,the searching unit 15 may search an illustration on the Internet withoutpreparing the illustration table. Alternatively, the informationprocessing system 3 may generate an illustration corresponding to a wordincluded in language information using the technology such as agenerative adversarial network (GAN).

The illustration may also include a 3D model or a three-dimensionalillustration. The 3D model or the three-dimensional illustration is avirtual three-dimensional object in a three-dimensional space. The usercan view the 3D model or the three-dimensional illustration from anyarbitrary direction within 360 degrees. The information processingsystem 3 may also convert the 3D model or the three-dimensionalillustration into an illustration on a two-dimensional plane.

Meeting Management Table

FIG. 8 is a conceptual diagram illustrating an example of meetingmanagement table. The storage unit 3000 includes a meeting managementstorage unit 3004, implemented by a meeting management table asillustrated in FIG. 8 . The meeting management table stores a pluralityof items such as a participant, a title (meeting name), a start date andtime, an end date and time, a place (location), and a password inassociation with a meeting ID of each meeting.

The participant item indicates a participant of the meeting.

The title item describes contents of the meeting, such as a name of themeeting or a topic of the meeting.

The start date and time item is a date and time when the meeting isscheduled to be started.

The end date and time item is a date and time when the meeting isscheduled to end.

The place item is a place where the meeting is held, which may berepresented by, for example, a name of a meeting room, a name of abranch office, a name of a building, etc.

The password item is authentication information for allowingparticipation in the meeting, and is distributed to the meetingparticipants in advance.

Functional Configuration of Terminal Apparatus

Referring back to FIG. 4 , a functional configuration of the terminalapparatus 1 is described according to the embodiment. The terminalapparatus 1 includes a first communication unit 11, a voice input unit(voice input) 12, a hand drafting input unit (hand drafting input) 13, amorphological analyzer 14, a searching unit 15, a first display controlunit 16, a second display control unit 17, an operation input unit(operation input) 18, an update notifying unit 19, a hierarchizationunit 20, a log storage processor 21, a playback unit 22, and a filteringunit 23. These units of the terminal apparatus 1 are functions or meansimplemented by or caused to function by operating one or more hardwarecomponents illustrated in FIG. 3 in cooperation with instructions of theCPU 501 according to the program loaded from the HD 504 to the RAM 503.The terminal apparatus 1 also includes a storage unit 1000 implementedby RAM 503 and HD 504 illustrated in FIG. 3 .

The first communication unit 11 communicates with the informationprocessing system 3 via a network. The first communication unit 11transmits speech data or hand-drafted data to the information processingsystem 3, and receives language information converted from the speechdata or hand-drafted data.

The voice input 12 receives input of speech collected at the microphone518, and acquires speech data as speech information based on thereceived speech. That is, the voice input 12 performs pulse codemodulation (PCM) on speech (that is, voice), which is an analog signal,and converts the speech into speech data which is digital data.

The hand drafting input 13 receives input of stroke data (a sequence ofcoordinate points), based on detected coordinates of a position where aninput device such as a pen is made in contact with the touch panel. Thehand drafting input 13 may receive input of stroke data (a sequence ofcoordinate points) in response to an operation with a mouse. The handdrafting input 13 connects a plurality of contact coordinates into acoordinate point sequence by interpolation, to generate stroke data.Stroke data is data displayed on a display based on a trajectory ofcoordinates of a stroke input with the input device. The stroke data maybe interpolated appropriately. Hand drafted data is data having one ormore pieces of stroke data.

The morphological analyzer 14 performs morphological analysis onlanguage information with the information processing system 3, anddecomposes the language information into words with parts of speech. Themorphological analysis is a process of decomposing language informationinto meaningful minimum units and determining a meaning, a part ofspeech, etc.

The searching unit 15 searches an illustration storage unit 1002 using acharacter string of a word decomposed by the morphological analyzer 14.The searching unit 15 acquires an illustration matching the characterstring, which is searched, from the illustration storage unit 1002.

The first display control unit 16 displays language information, a listof illustrations, menu icons, etc., in the conversation log area 101 ofthe information display screen 100. The second display control unit 17displays language information, a list of illustrations, illustrations,menu icons, etc., in the whiteboard work area 102 of the informationdisplay screen 100.

The operation input 18 receives various operations on the conversationlog area 101 or the whiteboard work area 102. Examples of such operationinclude pressing a menu icon, selecting content, editing content, movingcontent, and deleting content.

When there is a change in content in the whiteboard work area 102, theupdate notifying unit 19 notifies the information processing system 3 ofinformation related to the content in order to share the content withthe counterpart terminal apparatus 9. The change in content is detected,when there is any change to be shared for a particular content, such aslocation, size, color, etc. of the content.

The hierarchization unit 20 determines the positional relationship of aplurality of contents placed on the whiteboard work area 102 accordingto a hierarchical structure, and arranges the respective contents inhierarchy.

The log storage processor 21 stores, in a content log storage unit 1001,the language information in the conversation log area 101 in associationwith the content in the whiteboard work area 102. That is, the languageinformation displayed in the conversation log area 101 at a certain timeis associated with the content of the whiteboard work area 102.

When the user requests to playback the language information in theconversation log area 101 in order to review the discussions in themeeting, the playback unit 22 displays (playbacks) the content displayedin the whiteboard work area 102 in accordance with the languageinformation displayed in the conversation log area 101.

The filtering unit 23 filters the language information displayed in theconversation log area 101 by such as the user ID. This reduces thenumber of pieces of language information displayed in the conversationlog area 101, thus allowing the user to easily search for languageinformation of the user's interest.

Content Log Table

FIGS. 9A and 9B (FIG. 9 ) are a conceptual diagram illustrating anexample of content log table. The storage unit 1000 includes a contentlog storage unit 1001, implemented by a content log table as illustratedin FIGS. 9A and 9B. The content log table stores a status of theconversation log area 101 and a status of the whiteboard work area 102in a chronological order, each in association with a status ID.

The status ID item is identification information identifying a status ofthe conversation log area 101 and a status of the whiteboard work area102, which are associated.

The status of the conversation log area 101 indicates which languageinformation is displayed in the conversation log area 101 by a languageinformation ID of specific language information being displayed. Forexample, in a case where the status ID is s001, the language informationof t001 to t002 is displayed in the conversation log area 101. In thecorresponding whiteboard work area 102, the content indicated in thewhiteboard work area status field identified with the status ID s001 isdisplayed. Next, it is assumed that t003 is added to the conversationlog area 101. Accordingly, in the whiteboard work area 102 having astatus where the language information of t001 to t003 is displayed, thecontent indicated in the whiteboard work area status field identifiedwith the status ID of s002 is displayed. In this way, the status of theconversation log area 101 and the status of the whiteboard work area 102are associated with each other.

As described above, the content log table stores information regardingthe previous statuses of the whiteboard work area 102. Therefore, theterminal apparatus 1 is able to playback the whiteboard work area 102reflecting the previous status, in a similar manner as it were firstlydisplayed, rather than as an image such as a snapshot. However, theprevious status of the whiteboard work area 102 may be stored as animage such as a snapshot. Further, the entire information display screen100 may be recorded as a moving image.

The above-described information regarding the previous statuses, managedby the content log storage unit 1001, may be recorded at the informationprocessing system 3 and transmitted to the terminal apparatus 1.However, there may be a case where the number of pieces of languageinformation that can be displayed in the conversation log area 101differs depending on the terminal apparatus 1. In such a case, it ispreferable to store the pieces of language information for each terminalapparatus 1.

Illustration Table

The illustration storage unit 1002 included in the terminal apparatus 1may have the same structure as the illustration storage unit 3003included in the information processing system 3. In case the terminalapparatus 1 executes a Web application, information managed by theillustration storage unit 3003 of the information processing system 3 isdownloaded from the information processing system 3 to the terminalapparatus 1 via the network, as information to be managed by theillustration storage unit 1002. In case the terminal apparatus 1executes a native application, information to be managed by theillustration storage unit 1002 is stored at the native application inadvance. When the terminal apparatus 1 executes the native application,information to be managed by the illustration storage unit 1002 may bedownloaded.

Preferred Language Information Table

FIG. 10 is a conceptual diagram illustrating an example of preferredlanguage information table. The storage unit 1000 includes a preferredlanguage information storage unit 1003, which is implemented by apreferred language information table as illustrated in FIG. 10 . Thepreferred language information table stores the language informationpreferred by the user, selected from among the language information, inthe conversation log area 101. The preferred language information tablestores a list of language information IDs each being selected aspreferable by the user. The preferred language information table isoften different for each terminal apparatus 1, as each terminalapparatus 1 is operated by a different user.

Example of Information Display Screen

Next, referring to FIG. 11 , an information display screen 100 displayedby the terminal apparatus 1 is described according to the embodiment.FIG. 11 is an information display screen 100 in an initial state, whichis displayed by the terminal apparatus 1. The information display screen100 may be displayed using a Web application or a native application.

The information display screen 100 includes a conversation log area 101,a whiteboard work area 102, a tool display field 103, and a speechrecognition result display field 110.

The tool display field 103 displays various types of tools. The toolsinclude a content selection tool 111, a pen tool 112 for hand-draftinginput, a graphic tool 113 for drawing shapes, a text tool 114 forinputting characters, a lasso tool 115 for drawing a selection border,and the like. The above-described tools are just examples. The toolsalso include a display switch tool 116 for switching display orundisplay of the conversation log area 101. When the display switch tool116 is pressed, the conversation log area 101 is closed, and thewhiteboard work area 102 is displayed in larger size.

The conversation log area 101 displays the language informationconverted from the speech data in a chronological order.

When new language information is added to the conversation log area 101,the old language information moves from top to bottom. The languageinformation that has reached the bottom of the conversation log area 101is scrolled out. Even when scrolling out, the user can display theprevious language information by operating the scroll bar.

The speech recognition result display field 110 displays, therein,language information converted from speech data acquired by the voiceinput 12 of the terminal apparatus 1 in real time. The languageinformation displayed in the speech recognition result display field 110may also be shared with the counterpart terminal apparatus 9. For thelanguage information being displayed in the speech recognition resultdisplay field 110, an illustration is automatically displayed. Forexample, when the status where no speech is detected continues for apredetermined time period, the language information being displayed inthe speech recognition result display field 110 is transmitted to theinformation processing system 3. The information processing system 3transmits an instruction to display the converted language informationin the conversation log area 101 to each terminal apparatus 1, such thatthe language information is shared between different terminalapparatuses.

The user can arbitrarily input content to the whiteboard work area 102.That is, the user can use the whiteboard work area 102, as an electronicwhiteboard. Further, as will be described later, the user can copy thelanguage information displayed in the conversation log area 101 to thewhiteboard work area 102.

The user may adjust the positions or sizes of the conversation log area101 and the whiteboard work area 102. For example, the user may switchthe locations of the conversation log area 101 and the whiteboard workarea 102, between left and right of the screen. When the user increasesthe display size of the whiteboard work area 102, the conversation logarea 101 may be reduced in size accordingly. The same applies when theuser changes the display size of the conversation log area 101. Thewhiteboard work area 102 may be larger than the display 506 indisplayable size. In such case, the display 506 displays only a part ofthe whiteboard work area on the display 506 in response to an operationof the user.

Speech Recognition and Sharing of Language Information

Next, referring to FIG. 12 , operation of sharing language informationconverted from speech data, by each terminal apparatus 1 participatingin a meeting, is described according to an embodiment. FIG. 12 is asequence diagram illustrating processing of transmitting speech datafrom the terminal apparatus 1 to the information processing system 3,and processing to share language information converted from the speechdata at the information processing system 3 with the counterpartterminal apparatus 9, according to the embodiment. The terminalapparatus 1 and the counterpart terminal apparatus 9 participate in thesame meeting, for example, by each connecting to a URL transmitted bysuch as an e-mail invitation. The information processing system 3manages the plurality of terminal apparatuses 1 participating in thesame meeting using user IDs, IP addresses, or the like, of theparticipants. Alternatively, each user at the terminal apparatus 1 mayselect a particular meeting registered in the meeting management tableand input a preset password, to participate in the same meeting. In suchcase, the information processing system 3 manages the plurality ofterminal apparatuses 1 participating in the same meeting by using userIDs, IP addresses, etc. of the participants.

S101: The voice input 12 of the terminal apparatus 1 receives input ofvoices (speech) uttered by the user.

S102: The voice input 12 transmits speech data, which is digital dataconverted from voices, to the information processing system 3 via thefirst communication unit 11.

S103: The second communication unit 31 of the information processingsystem 3 receives the speech data. The converter 32 performs speechrecognition on the speech data to convert the speech data into languageinformation.

S104, S105: The second communication unit 31 transmits the languageinformation to the terminal apparatus 1 and the counterpart terminalapparatus 9 participating in the meeting. Since the speech recognitionis performed in real time, the content of the speech may not bedetermined at this time.

S106, S107: The first communication unit 11 of the terminal apparatus 1and the first communication unit 11 of the counterpart terminalapparatus 9 each receive the language information. At each of theterminal apparatus 1 and the counterpart terminal apparatus 9, the firstdisplay control unit 16 displays the undetermined language informationin the speech recognition result display field 110, as the languageinformation is received.

S108: The conversation manager 33 stores the language information ID(newly assigned number), the language information, the input time, thespeaker (the user who has logged in), and the number of likes (theinitial value is 0) in the conversation table, based on a detection of abreak of the utterance by the user. For example, the break time isdetected when a period of time for not receiving any voices (time forsilence) exceeds a preset value.

S109, S110: The second communication unit 31 transmits the languageinformation ID, the language information, and the input time to theterminal apparatus 1 and the counterpart terminal apparatus 9.

S111, S112: The first communication unit 11 of the terminal apparatus 1and the first communication unit 11 of the counterpart terminalapparatus 9 each receive the language information ID, the languageinformation, and the input time. At each of the terminal apparatus 1 andthe counterpart terminal apparatus 9, the first display control unit 16additionally displays new language information in the conversation logarea 101. In this example, the input time may not be transmitted. Forexample, the input time, when transmitted, is used for highlighting thelanguage information corresponding to the content selected in thewhiteboard work area 102 (see FIG. 24 ).

As display of the conversation log area 101 changes, the log storageprocessor 21 acquires the language information ID of the languageinformation currently displayed in the conversation log area 101, as thestatus of the conversation log area 101. In addition, the log storageprocessor 21 acquires the content data being displayed in the whiteboardwork area 102, as the status of the whiteboard work area 102.Information on such content data may be acquired from the informationprocessing system 3 or may be acquired from the content being displayedon the terminal apparatus 1. The log storage processor 21 stores thestatus of the conversation log area 101 and the status of the whiteboardwork area 102 in association with each other in the content log storageunit 1001.

Copying of Language Information from Conversation Log Area to WhiteboardWork Area

Next, referring to FIGS. 13A to 13D, a display example of content in thewhiteboard work area 102 using the language information in theconversation log area 101 is described according to the embodiment.FIGS. 13A to 13D are schematic diagrams illustrating processing ofcopying language information from the conversation log area 101 to thewhiteboard work area 102, performed by the terminal apparatus 1, inresponse to a user operation, according to the embodiment.

FIG. 13A illustrates the conversation log area 101 displaying somelanguage information. While the whiteboard work area 102 does notdisplay any content in this example, the whiteboard work area 102 maydisplay any content. The user selects (taps or clicks) any languageinformation (for example, language information 121) in the conversationlog area 101 for use in the whiteboard work area 102. The selectedlanguage information 121 is decomposed into character strings one by oneby morphological analysis. The character strings correspond to morphemesbased on user-defined rules. The language information 121 may bedecomposed into characters, such as only Chinese characters, if thelanguage information 121 is written in Japanese. The searching unit 15searches the illustration storage unit 1002 using the decomposedcharacter strings.

As illustrated in FIG. 13B, the first display control unit 16 displays alist 122 of illustrations (illustration list 122) related to theselected language information 121, near (in this example, below) theselected language information 121 in the conversation log area 101. Theillustration list 122 lists thumbnails of a plurality of illustrations(there may be only one illustration). The user can select anillustration to be displayed in the whiteboard work area 102 from theillustration list 122. The user may drag and drop the selectedillustration into the whiteboard work area 102. Alternatively, the usermay simply select an illustration from the illustration list 122 tocause the selected illustration be displayed in the whiteboard work area102.

The user may select an illustration from the illustration list 122,which is generated in response to a speech of another user, in additionto or in alternatively to the illustration list 122 generated based onthe speech of the user himself or herself. With this selection, theother user knows that his or her speech has been referred to, and canfeel that he or she actively participates in the meeting. It is assumedthat the user does not select an illustration in FIG. 13B.

Further, the terminal apparatus 1 may store illustrations correspondingto the language information displayed in the speech recognition resultdisplay field 110 at least temporarily, and display the illustrations inthe form of the illustration list 122. This can shorten a period of timefrom the time when the user selects the language information 121, to thetime when the illustration list 122 is displayed.

As illustrated in FIG. 13C, the user selects a character string 123which is a part of the language information, and drags and drops a copyof the character string 123 (hereinafter referred to as a characterstring 126) from the conversation log area 101 to the whiteboard workarea 102. FIG. 13D illustrates the character string 126 displayed in thewhiteboard work area 102. Through this processing, the user canunderstand how his or her speech affects the content of the whiteboardwork area 102.

The user can also cause the character string 123 to be displayed on thewhiteboard work area 102, by clicking on a part of or entire characterstring 126 and then clicking on the whiteboard work area 102. Theillustrations in the illustration list 122 may also be displayed in thewhiteboard work area 102, by either dragging and dropping or clicking bythe user.

As described above, input operation to the whiteboard work area 102 isreduced, as the user can use the language information in theconversation log area 101. The user can further move the characterstring 126 to a different position, or rotate the character string by anarbitrary angle.

As illustrated in FIG. 13A, a like button 124 and a favorite button 125are displayed for each language information in the conversation log area101. The like button 124 and the favorite button 125 may be displayed atall times, or may be displayed when the user selects particular languageinformation or performs mouse-over operation on particular languageinformation. When the user presses the like button 124, the languageinformation ID and information indicating that the like button has beenpressed are transmitted to the information processing system 3, and thenumber of likes in the conversation table is increased by one. When theuser presses the favorite button 125, the language information ID isadded to the preferred language information table of the terminalapparatus 1 operated by the user.

As illustrated in FIG. 13D, when content is added to the whiteboard workarea 102, the update notifying unit 19 transmits the type, page,coordinates, size, font size, and language information ID of the contentto the information processing system 3 via the first communication unit11. Since the content manager 34 transmits one record of the contenttable, which is newly added, to the counterpart terminal apparatus 9,the terminal apparatus 1 can share the current status of the whiteboardwork area 102 with the counterpart terminal apparatus 9.

When the illustration list 122 as illustrated in FIGS. 13A to 13D isdisplayed, the first display control unit 16 may determine a displayorder of the thumbnails of the illustration list 122, in accordance withthe language information 121 from which each illustration is derived.

FIGS. 14A and 14B are diagrams for explaining the display order ofthumbnails of illustrations, which are determined based on languageinformation. FIGS. 14A and 14B illustrate how the display order of thethumbnails changes depending on different language information.

The first display control unit 16 performs morphological analysis on thelanguage information, and searches the illustration table sequentiallyin an order from the first word to the last word, among a plurality ofwords obtained by the morphological analysis. Specifically, the firstdisplay control unit 16 extracts an illustration suitable to thesearching word, and displays the illustration as an individual thumbnailimage in the illustration list. Assuming that a plurality ofillustrations each match one word in the illustration table, if thefirst display control unit 16 arranges thumbnails of all illustrationsin the illustration list, a display area of the illustration list isfilled with the thumbnails of the illustrations for only one word. Inorder to prevent this, the first display control unit 16 displays onlyone thumbnail of illustrations for one word, for one sentence oflanguage information (one sentence being an example of languageinformation in a predetermined unit). When one thumbnail of theillustration is displayed for one sentence, the first display controlunit 16 returns to the beginning of the same sentence, and continues todisplay another set of thumbnails for the same words.

FIG. 14A illustrates a illustration list 230, displayed for languageinformation 239 of “animal lover person (a person who loves animals)”,uttered by the user. Words detected by the morphological analysis are“animal”, “love(r)”, and “person” in this order. Since the first displaycontrol unit 16 displays only one thumbnail for one word, in FIG. 14A, athumbnail 231 corresponding to the word “animal”, a thumbnail 232corresponding to the word “love(r)”, and a thumbnail 233 correspondingto the word “person” are displayed in this order. When the thumbnail ofthe illustration has been displayed for one sentence, the first displaycontrol unit 16 returns to the beginning of the same sentence, andcontinues to display another set of thumbnails for the same words.Therefore, in FIG. 14A, a thumbnail 234 corresponding to the word“animal”, a thumbnail 235 corresponding to the word “love(r)”, athumbnail 236 corresponding to the word “person”, are displayed in thisorder. Similarly, a thumbnail 237 corresponding to the word “animal”,and a thumbnail 238 corresponding to the word “love(r)” are displayed inthis order.

FIG. 14B illustrates an illustration list 240 of illustrations displayedfor the language information 249 “people and animals, (I) love (I lovepeople and animals)”, uttered by the user. Words detected by themorphological analysis are “people”, “animals”, and “love” in thisorder. The order of appearance of words is different from that in FIG.14A. Therefore, in FIG. 14B, a thumbnail 241 corresponding to the word“people”, a thumbnail 242 corresponding to the word “animals”, athumbnail 243 corresponding to the word “love”, a thumbnail 244corresponding to the word “people”, a thumbnail 245 corresponding to theword “animals”, a thumbnail 246 corresponding to the word “love”, athumbnail 247 corresponding to the word “people”, and a thumbnail 248corresponding to the word “animals” are displayed in this order.

Display of thumbnails in the illustration list can also be controlled,in the case of displaying the words in the whiteboard work area 102, ina substantially similar manner (an illustration list 142 in FIG. 17B).

The first display control unit 16 may display a plurality of thumbnailsfor one word, if the display area of the illustration list is not filledwith only thumbnails of illustrations for the single word. Further,morphological analysis may be collectively performed on a plurality ofsentences (or a plurality of predetermined units for analysis), and onlyillustrations common to the words detected as a result of the analysismay be displayed.

FIG. 15 is a sequence diagram illustrating processing of copyinglanguage information from the conversation log area 101 to thewhiteboard work area 102, performed by the terminal apparatus 1, inresponse to a user operation, in the above-described example case ofFIGS. 13A to 13D.

S1: As the user selects the language information (in this example, thelanguage information 121) in the conversation log area 101, theoperation input 18 receives the selection by the user.

S2: The morphological analyzer 14 performs morphological analysis on thelanguage information 121, and decomposes the language information 121into character strings. The morphological analysis does not have to beperformed, such that the morphological analyzer 14 may extract onlyChinese characters from sentences, if the language information iswritten in Japanese written in mixture of “Kana” characters and Chinesecharacters. The searching unit 15 searches the illustration storage unit1002 for a keyword that matches each character string. The first displaycontrol unit 16 displays the illustration list 122, which listsillustrations associated with each keyword used for the search, near theselected language information 121.

S3: The user drags and drops a copy of a part of the character string123 of the language information, to the whiteboard work area 102, as thecharacter string 126. The operation input 18 receives theabove-described user operation, and the second display control unit 17displays the character string 126 at a destination where the user hasdropped. The character string 126 is an example of content based on auser operation on the language information.

S4: The update notifying unit 19 detects that the status of thewhiteboard work area 102 has been changed. The update notifying unit 19transmits a content update notification (content type, character string126, page, coordinates, size, font size, and language information ID) tothe information processing system 3 via the first communication unit 11.

S5: The second communication unit 31 of the information processingsystem 3 receives the content update notification. The content manager34 assigns a content ID to the copied content and stores the content IDin the content table together with items of the content updatenotification.

S6: The second communication unit 31 of the information processingsystem 3 transmits the content ID to the terminal apparatus 1.

S7: As a change has occurred in the content table, the content manager34 transmits a content sharing request (content ID, content type,character string 126, page, coordinates, size, font size, and languageinformation ID) to the counterpart terminal apparatus 9 participating inthe same meeting.

S8: The first communication unit 11 of the counterpart terminalapparatus 9 receives the content sharing request. The second displaycontrol unit 17 displays the character string 126 at the positionindicated by the coordinates, in a font having the font size beingspecified.

Full Text Copy of Language Information from Conversation Log Area toWhiteboard Work Area

Next, referring to FIGS. 16A to 16D, an example display of content inthe whiteboard work area 102 using language information in theconversation log area 101 is described according to the embodiment.FIGS. 16A to 16D are schematic diagrams illustrating processing ofcopying full text of language information from the conversation log area101 to the whiteboard work area 102, performed by the terminal apparatus1, in response to a user operation, according to the embodiment. In thedescription referring to FIG. 16 , for simplicity, the processing thatare different from the processing of FIG. 15 are described.

The illustration of FIG. 16A is the same as that of FIG. 13A. In FIG.16B, the entire copy of one language information 131 (hereinafterreferred to as language information 132) is dragged. In a case where theuser hovers a mouse pointer, the first display control unit 16 does notmove the language information in the conversation log area 101. Thisprevents the language information 131 to be copied, from moving, thuspreventing operability from decreasing.

In FIG. 16C, one language information 132 is continuously dragged. FIG.16D illustrates a state in which the user has dropped the languageinformation 132. In this way, the user can copy the entire languageinformation 131 in the conversation log area 101 to the whiteboard workarea 102. As will be described later, since the user can also performmorphological analysis on the language information in the whiteboardwork area 102, the user can use the entire character strings included inthe language information 131, without dragging and dropping only adesired character string.

The processing of FIG. 16 may be the same as that of FIG. 15 except thatthe number of character strings copied to the whiteboard work area 102are different.

Displaying Illustration Based on Language Information in Whiteboard WorkArea

Referring next to FIGS. 17A to 17C, a display example of an illustrationbased on language information in the whiteboard work area 102 isdescribed according to the embodiment. FIGS. 17A to 17C are diagramsillustrating a display example of an illustration, searched based on thelanguage information copied to the whiteboard work area 102.

During discussions in the meeting, the user directly inputs languageinformation into the whiteboard work area 102, or places languageinformation selected from the conversation log area 101 in thewhiteboard work area 102. If the illustration related to the languageinformation is displayed along the language information, the content ofthe language information becomes easy to understand.

As illustrated in FIG. 17A, the user selects (taps or clicks) thelanguage information 132 in the whiteboard work area 102. The selectedlanguage information 132 is decomposed into character strings one by oneby morphological analysis. The character strings correspond to morphemesbased on user-defined rules. The language information 121 may bedecomposed into characters, such as only Chinese characters, if thelanguage information 121 is written in Japanese. The searching unit 15searches the illustration storage unit 1002 using the decomposedcharacter strings.

As illustrated in FIG. 17B, the second display control unit 17 displaysa list 142 of illustrations (illustration list 142) related to thelanguage information and menu icons 141, near (in this example, belowand above) the selected language information 132 in the whiteboard workarea 102. The menu icons 141 include a user ID display icon 141 a, anicon 141 b for setting color of language information or a color of aframe, a language information centering icon 141 c, a morphologicalanalysis icon 141 d, and a language information fixing icon 141 e, etc.The above-described icons are merely examples.

The user can select an illustration to be displayed in the whiteboardwork area 102 from the illustration list 142. As illustrated in FIG.17C, the second display control unit 17 displays selected illustration143. The size of the illustration 143 is assumed to be determined inadvance, but can be changed as appropriate by the user.

By selecting the illustration 143 from the displayed illustration list142, the user can smoothly add visual information to help convey thecontent during the meeting, without searching for a suitableillustration by himself or herself. The user can move the illustration143 to another position, or rotate the illustration 143 by an arbitraryangle.

FIG. 18 is a sequence diagram illustrating processing of displaying anillustration based on language information of the whiteboard work area102, performed by the terminal apparatus 1, in response to a useroperation, in the above-described case referring to FIGS. 17A to 17C.

S11: As the user performs an operation on particular languageinformation (in this example, the language information 132) in thewhiteboard work area 102 to make a selection, the operation input 18receives the selection by the user.

S12: The morphological analyzer 14 performs morphological analysis onthe language information 132, and decomposes the language information132 into character strings. The morphological analysis does not have tobe performed, such that the morphological analyzer 14 may extract onlyChinese characters from sentences, if the language information iswritten in Japanese written in mixture of “Kana” characters and Chinesecharacters. The searching unit 15 searches the illustration storage unit1002 for a keyword that matches each character string. The seconddisplay control unit 17 displays the illustration list 142, which listsillustrations associated with each keyword used for the search, and themenu icons 141, near the selected language information 132.

S13: The user performs an operation on a particular illustration in theillustration list 142 to select the illustration. The operation input 18receives the operation of selecting the illustration.

S14: The second display control unit 17 displays the selectedillustration 143, near the language information 132 in the whiteboardwork area 102. The illustration list 142 and the illustration 143 areexamples of content based on a user operation on the languageinformation.

S15: The update notifying unit 19 detects that the status of thewhiteboard work area 102 has been changed. The update notifying unit 19transmits a content update notification (content type, illustration,page, coordinates, size, and language information ID) to the informationprocessing system 3 via the first communication unit 11.

S16: The second communication unit 31 of the information processingsystem 3 receives the content update notification. The content manager34 assigns a content ID to the selected content, and stores the contentID in the content table together with items of the content updatenotification.

S17: The second communication unit 31 of the information processingsystem 3 transmits the content ID to the terminal apparatus 1.

S18: As a change has occurred in the content table, the content manager34 transmits a request for sharing content (content ID, content type,illustration, page, coordinates, size, and language information ID) tothe counterpart terminal apparatus 9 participating in the same meeting.

S19: The first communication unit 11 of the counterpart terminalapparatus 9 receives the request for sharing the content. The seconddisplay control unit 17 displays the illustration 143 at a positionindicated by the coordinates being specified.

Decomposition of Language Information into Character Strings

Next, referring to FIGS. 19A to 19C, an example of decomposition oflanguage information into character strings, in the whiteboard work area102, is described according to the embodiment. FIGS. 19A to 19C arediagrams for explaining decomposition of language information copied tothe whiteboard work area 102 into character strings.

In copying the language information from the conversation log area 101to the whiteboard work area 102, it is cumbersome for the user toperform operation of selecting and cutting out a part of the languageinformation, while having conversations during the meeting. In view ofthis, the terminal apparatus 1 performs morphological analysis on thelanguage information, decomposes the language information into characterstrings, and displays the character strings.

The screens illustrated in FIGS. 19A and 19B are substantially the sameas those illustrated in FIGS. 17A and 17B. In FIG. 19B, the user pressesthe morphological analysis icon 141 d. Accordingly, as illustrated inFIG. 19C, the morphological analyzer 14 performs morphological analysison the language information 132 in the whiteboard work area 102. Thesecond display control unit 17 deletes the language information 132 fromthe screen, and displays one or more character strings 144 to 146, basedon a result of morphological analysis. The arrangement order of thecharacter strings 141 to 146 is not limited to the one illustrated inFIG. 19C. The displayed character strings 144 to 146 are, for example,nouns and verbs. The character strings to be display may further includeadjectives. It is preferable that the user can select a part of speechto be displayed. Further, the language information 132 may not bedeleted from the screen.

The size of each of the character strings 144 to 146 may be the same asthat of the original language information. Alternatively, the size ofeach of the character strings 144 to 146 may be determined according tothe number of characters of the character strings 144 to 146. Thedisplay positions of the character strings 144 to 146 are arranged inthe order of appearance of the character strings 144 to 146, withreference to the upper left corner of the circumscribed rectangle of thelanguage information 132 as an origin. When a size of the characterstrings 144 to 146 being arranged exceeds a size determined by thecircumscribed rectangle of the language information 132, such that thecharacter strings 144 to 146 extend to the right by a certain amount ormore, the second display control unit 17 makes a new row to display apart of the character string that does not fit.

The character strings 144 to 146 having been generated throughdecomposition as described above are still associated with the languageinformation ID of the language information in the conversation log area101.

In FIGS. 19A to 19C, the user presses the morphological analysis icon141 d to execute the morphological analysis on the language information132 copied to the whiteboard work area 102. Alternatively, as the userselects one piece of language information in the conversation log area101, the entire text (language information 132) may be placed in thewhiteboard work area 102. The morphological analysis may beautomatically executed on the entire language information 132 todecompose the language information 132 into character strings. That is,the user is able to instruct to decompose the language information 131into the character strings 144 to 146, only by pressing the languageinformation 131 in the conversation log area 101. In this case, the usermay select whether to use the language information 132 or the characterstrings 144 to 146.

Furthermore, the second display control unit 17 can also automaticallydisplay illustrations of the character strings 144 to 146, which aredisplayed after decomposition, in the whiteboard work area 102. That is,the user is able to display illustrations of the character strings 144to 146, only by pressing the language information 131 in theconversation log area 101.

FIG. 20 is a sequence diagram illustrating processing of decomposinglanguage information in the whiteboard work area 102 into characterstrings, performed by the terminal apparatus 1, in response to a useroperation, in the example case referring to FIGS. 19A to 19C.

S21: As the user selects the language information 132 in the whiteboardwork area 102, the operation input 18 receives the selection by theuser.

S22: The morphological analyzer 14 performs morphological analysis onthe language information 132, and decomposes the language information132 into words. In this example, a word is an example of morpheme. Themorphological analysis does not have to be performed, such that themorphological analyzer 14 may extract only Chinese characters fromsentences, if the language information is written in Japanese written inmixture of “Kana” characters and Chinese characters. The searching unit15 searches the illustration storage unit 1002 for a keyword thatmatches each word. The second display control unit 17 displays theillustration list 142, which lists illustrations associated with eachkeyword used for the search, and the menu icons 141, near the selectedlanguage information 132.

S23: The user selects the morphological analysis icon 141 d from themenu icons 141. The operation input 18 receives selection of themorphological analysis icon 141 d.

S24: The morphological analyzer 14 executes morphological analysis onthe selected language information 132.

S25: The second display control unit 17 deletes the selected languageinformation 132 from the screen, and displays the character strings 144to 146 in the whiteboard work area 102. At least one of the characterstrings 144 to 146 is an example of content based on a user operation onthe language information.

S26: The update notifying unit 19 detects that the status of thewhiteboard work area 102 has been changed. The update notifying unit 19transmits a content update notification 1 that requests deletion ofparticular content (content ID, delete), and a content updatenotification 2 that requests to add particular content (content type,character strings 144 to 146, page, coordinates, size, font size,language information ID) to the information processing system 3 via thefirst communication unit 11.

S27: The second communication unit 31 of the information processingsystem 3 receives the content update notification. The content manager34 deletes the content specified by the content ID in the content updatenotification 1. The content manager 34 further stores the contents, suchas the character strings 144 to 146, in the content table, as specifiedby the content update notification 2.

S28: The second communication unit 31 of the information processingsystem 3 transmits the content IDs of the character strings 144 to 146to the terminal apparatus 1.

S29: As a change has occurred in the content table, the content manager34 transmits a content sharing request 1 (content ID, delete), and acontent sharing request 2 (content ID, content type, character strings144 to 146, page, coordinates, size, font size, language informationID), to reflect the change, to the counterpart terminal apparatus 9participating in the same meeting.

S30: The first communication unit 11 of the counterpart terminalapparatus 9 receives the content sharing request. The second displaycontrol unit 17 deletes from the screen (not displays) the languageinformation 132 specified by the content ID in the content sharingrequest 1. The second display control unit 17 displays the characterstrings 144 to 146 at the positions indicated by the coordinates, eachin a font having the specified font size, according to the contentsharing request 2.

Hierarchization of Contents

Next, referring to FIGS. 21A to 21C, hierarchization of a plurality ofcontents displayed in the whiteboard work area 102 is describedaccording to the embodiment. FIGS. 21A to 21C are diagrams forexplaining hierarchization of a plurality of contents displayed in thewhiteboard work area 102.

In the meeting, a user may want to display information in hierarchyaccording to a certain rule. In such case, it is cumbersome for the userto manually arrange the information in hierarchy. In view of this, thehierarchization unit 20 arranges the contents based on the positionalrelationship between the plurality of contents, which are displayed inthe whiteboard work area 102.

As illustrated in FIG. 21A, the user firstly makes a group based oncontents 151 to 154 subjected to hierarchization, using the contentselection tool 111. In FIG. 21A, the contents 151 to 154 subjected tohierarchization are character strings, but the contents subjected tohierarchization may be illustrations. Through grouping, a frame 150 anda hierarchy button 159 are displayed, and the user presses (taps orclicks) the hierarchy button 159. The user may further specify ahierarchical structure (such as the number of layers, the number ofnodes).

FIG. 21B illustrates a list of hierarchical structures 156 a to 156 c,according to the embodiment. The user can select a desired one of thehierarchical structures 156 a to 156 c, and further edit, for example,to increase or decrease the number of nodes (A to F in FIG. 21B).

The hierarchization unit 20 arranges the plurality of selected contents151 to 154 to the nodes of the hierarchical structure that is selected,respectively, based on the positional relationship between the groupedcontents. Processing of hierarchization based on the positionalrelationship is described. In an example case of pyramid structure, thecontent 151 at the highest position (the y coordinate is small), amongthe contents 151 to 154, is placed at the node A, which is the top ofthe hierarchical structure. The remaining grouped contents are placed tothe nodes of the hierarchical structure, respectively, in an order fromthe upper left to the lower right.

An example case where the hierarchical structure 156 b in FIG. 21B isselected is described. The hierarchization unit 20 places the content151 at the highest position in the whiteboard work area 102 of FIG. 21A,to the node A. The hierarchization unit 20 determines priorities of theremaining contents 152 to 154 based on the position coordinates (x, y)of the contents 152 to 154. For example, the hierarchization unit 20gives a higher priority to the content with a total of the x coordinateand the y coordinate being smaller. The hierarchization unit 20 arrangesthe contents 152 to 154 to the nodes B, C, and D in descending order ofpriority.

FIG. 21C illustrates grouped contents 151 to 154 after hierarchization.Below the content 151, the contents 152 to 154 each being connected tothe content 151 by a line 155 are aligned. As described above, the usercan arrange the contents 151 to 154, each based on the languageinformation copied from the conversation log area 101, in hierarchy,with simple operation.

Although the hierarchical structure is selected by the user in thedescription of FIG. 21B, the positional relationship between the groupedcontents and the hierarchical structure may be modeled by machinelearning. For example, a model may be generated using deep learningtechnique, as a model corresponding to the number of pieces of content.Specifically, in such model, coordinates of one piece of content areinput, and a node position (a number or an ID indicating a nodeposition) at which the content is placed by a user is output as acorrect answer. This model outputs a probability of being placed at eachnode in response to input of coordinates of each content.

FIG. 22 is a block diagram illustrating a functional configuration of anode position output 220 based on machine learning, as described above.As illustrated in FIG. 22 , the node position output 220 may include acontent acquisition unit 221, a training data storage unit 222, amachine learning unit 223, a learned model storage unit 224, and aninference unit 225. Each of the functional units is described below.

The content acquisition unit 221 acquires the content, which has beenmoved from such as the language information displayed in theconversation log area 101, to the whiteboard work area 102, eitherautomatically or in response to a user operation.

Learning Phase:

The training data storage unit 222 stores training data used for machinelearning. The training data in the training data storage unit 222includes content (input), a hierarchical structure, and a node position(output) in the hierarchical structure, which are acquired andaccumulated by the content acquisition unit 221 for a certain period oftime.

The machine learning unit 223 generates a learned model for deriving anode position to be output from the received content. Specifically, themachine learning unit 223 performs machine learning using training data,and generates a learned model. The training data is a set of receivedcontent data as input data, and a node position selected after thecontent is received as output data. The machine learning unit 223further stores the generated learned model in the learned model storageunit 224.

The learned model storage unit 224 stores the learned model generated bythe machine learning unit 223.

Inference Phase:

The inference unit 225 acquires the current content, and infers the nodeposition where the content is placed. Specifically, the inference unit225 acquires the content from the content acquisition unit 221. Theinference unit 225 further inputs the content to the learned model inthe learned model storage unit 224, and outputs the node position to beselected next.

In the learning phase described above, the machine learning unit 223 mayacquire and learn user information such as a login ID of the user inaddition to the content. In this case, the machine learning unit 223 canfurther learn the node position according to the user who has input thecontent.

If the terminal apparatus 1 includes the node position output 220, as afunction of the hierarchization unit 20, the user of the terminalapparatus 1 can obtain, by selecting a particular content, ahierarchization result output based on learning of selection of theuser. Further, a provider of the visual communication system 5 maypreviously perform learning, using the node position output 220, oninput data by another particular user. The resultant learned model isprovided to the user via the hierarchization unit 200, such that theuser of the terminal apparatus 1 can obtain output of contents inhierarchy, which are appropriately generated based on the learningresult that is previously prepared.

FIG. 23 is a sequence diagram illustrating hierarchization of languageinformation in the whiteboard work area 102, performed by the terminalapparatus 1, in response to a user operation, according to theembodiment.

S41: When the user selects (groups) a plurality of contents 151 to 154in the whiteboard work area 102, the operation input 18 receives theselection by the user.

S42: When the user presses the hierarchy button 159 and further selectsa hierarchical structure, the operation input 18 receives the selectionof hierarchical structure.

S43: The hierarchization unit 20 arranges the plurality of selectedcontents 151 to 154, for which the selection is received at S41, inhierarchy, in accordance with the hierarchical structure selected atS42. Accordingly, information regarding the hierarchization (“hierarchyinformation”) is generated. The hierarchy information indicates how eachcontent is hierarchized. In the example of FIG. 21C, the hierarchyinformation includes the following.

Vertex Node “Ice Breaker”, Child Nodes of the Vertex Node“Self-Introduction”, “Word Chain”, “Yamanote-Line Game”

A case where the number of layers is three or more can be expressed inthe same manner. Each content in the hierarchy information is actuallyrepresented by the content ID of each of the contents 151 to 154.

S44: The update notifying unit 19 detects a change in the status of thewhiteboard work area 102. The update notifying unit 19 transmits acontent update notification (grouping, content ID, coordinates, size,font size, hierarchy information, and language information ID) to theinformation processing system 3 via the first communication unit 11.Some information, such as size and font size, does not have to beprovided. In this example, the grouping means information indicatingthat a plurality of contents, specified by the content IDs, are grouped.

S45: The second communication unit 31 of the information processingsystem 3 receives the content update notification. The content manager34 stores, for each content, the coordinates of the content specified bythe content ID in the content table.

S46: The content manager 34 stores the hierarchical structure, so thatthe state in which the contents are made in hierarchy is stored. Thecontent manager 34 may set an ID to the hierarchy information.

S47: As a change has occurred in the content table, the content manager34 transmits a request for sharing content (grouping, content ID,coordinates, hierarchy information, and language information ID) to thecounterpart terminal apparatus 9 participating in the same meeting.

S48: The first communication unit 11 of the counterpart terminalapparatus 9 receives the request for sharing content. The second displaycontrol unit 17 arranges the plurality of character strings eachidentified with the content ID in hierarchy based on the hierarchyinformation, and displays the character strings at the positions eachspecified by the coordinates.

Reference to Conversation Log Area by Content of Whiteboard Work Area

Next, referring to FIGS. 24A to 24C, processing of referring to theconversation log area 101 by the contents of the whiteboard work area102 is described according to the embodiment. FIGS. 24A to 24Cillustrate the language information in the conversation log area 101,which is highlighted when the content in the whiteboard work area 102 isselected.

As the discussion progresses, it would be helpful to review at whichscene during the meeting each content in the whiteboard work area 102was generated. The first display control unit 16 displays the languageinformation associated with the content selected in the whiteboard workarea 102, with emphasis in the conversation log area 101.

First, as illustrated in FIG. 24A, the user selects a particular contentin the whiteboard work area 102. In this example, it is assumed that theuser selects (taps or clicks) the illustration 161. Accordingly, asillustrated in FIG. 24B, the second display control unit 17 displays arelated information check icon 162 near the illustration 161. The userpresses the related information check icon 162. Accordingly, asillustrated in FIG. 24C, the first display control unit 16 displays thelanguage information 163 having the language information ID of theselected illustration 161 in the conversation log area 101 withemphasis. If the language information 163 is scrolled out, the firstdisplay control unit 16 scrolls back to display the language information163 in the vicinity of the center of the conversation log area 101 withemphasis.

The selected illustration 161 may not have a language information ID, ifthe user directly inputs the illustration 161 to the whiteboard workarea 102. In such case, the first display control unit 16 displays thelanguage information with emphasis, which has been input at the date andtime closest to the time when the selected illustration 161 is input (orat the date and time during a certain period of time from the date andtime when the selected illustration 161 is input).

In this example, displaying with emphasis means to display thecharacters in bold font, display the characters in different color, orscrolling a display area to make the corresponding characters visible.

FIG. 25 is a sequence diagram illustrating processing of displaying thelanguage information corresponding to the content in the whiteboard workarea 102, with emphasis, performed by the terminal apparatus 1, inresponse to a user operation, in the example referring to FIGS. 24A to24C.

S51: As the user selects the content (illustration 161) in thewhiteboard work area 102, the operation input 18 receives the selectionby the user.

S52: The second display control unit 17 displays the related informationcheck icon 162 for the selected illustration 161.

S53: When the illustration 161 selected in the content table has thelanguage information ID, the first display control unit 16 displays thelanguage information 163 identified with the language information ID ofthe selected illustration 161, in the conversation log area 101, withemphasis.

S54: When the illustration 161 selected in the content table does nothave the language information ID, the first display control unit 16displays the language information associated with an input time, whichis close to the time when the selected illustration 161 is input anddisplayed, in the conversation log area 101, with emphasis. The timethat is close to the time when the selected content is input anddisplayed, may be an input time within a predetermined period of timefrom a time immediately before the time when the selected content isinput and displayed. This is because the content derived from thelanguage information in the conversation log area 101 is arranged in thewhiteboard work area 102 only after the language information from whichthe content is derived is displayed. The predetermined period of timefrom the time immediately before the time when the selected content isinput and displayed may be approximated as a time, at the maximum, fromthe time when one piece of language information is displayed to the timewhen such language information is scrolled out. For example, suchpredetermined period of time may be set to about ½ to ¼ of such maximumvalue.

In this example of displaying the language information with emphasis,the status of the conversation log area 101 is not shared with thecounterpart terminal apparatus 9. If the status of the conversation logarea 101 is shared, the language information may scroll, against theintension of the user viewing the conversation log area 101 at thecounterpart terminal apparatus 9. Under some cases where a plurality ofusers are to review the meeting together, the status of the conversationlog area 101 may be shared with the counterpart terminal apparatus 9according to the settings by the user. That is, whether to share thestatus of the conversation log area 101 with emphasis, may be determinedaccording to the user preference, or a preset condition.

Filtering Language Information in Conversation Log Area

The number of pieces of language information in the conversation logarea 101 increases as the discussion progresses, and it becomesdifficult to review. Therefore, the first display control unit 16 has afunction of filtering the language information of the conversation logarea 101. Examples of filtering method are as follows.

One method selects language information using a user name (or user ID)of a particular user, so that language information is selected based ona speaker who gives the language information.

Another method selects language information in the conversion log area101, having a number of times of likes being pressed, which exceeds acertain number. Another method selects language information, which isregistered in a preferred language table that stores languageinformation preferred by a particular user. Another method selectslanguage information, based on input time (utterance time) of thelanguage information to the conversation log area 101.

Further, at the time of filtering, it is also effective that the seconddisplay control unit 17 limits the content to be displayed in thewhiteboard work area 102 to only the content derived from the content inthe conversation log area 101 having been selected as described above.The second display control unit 17 displays only the content associatedwith the language information ID of the filtered language information,and does not display the other content.

Alternatively, the second display control unit 17 may cause the content,which is not associated with the language information ID of the filteredlanguage information, to be hardly recognizable, for example, bylowering saturation or transmittance of such content.

FIG. 26 is a flowchart illustrating processing of filtering languageinformation in the conversation log area 101, performed by the terminalapparatus 1, according to the embodiment.

First, the operation input 18 of the terminal apparatus 1 receives arequest for starting execution of filtering (S61). For example, theinformation display screen 100 may display, therein, a button forexecuting filtering. The operation input 18 also receives selection of afiltering method, for example, by a radio button or the like.

The filtering unit 23 performs filtering on all language information inthe conversation log area 101 (S62).

The first display control unit 16 displays only the language informationto which the filtering is applied, in the conversation log area 101(S63). Language information that cannot be displayed in the conversationlog area 101 can be displayed by scrolling.

The second display control unit 17 displays only the content associatedwith the language information ID of the filtered language information inthe whiteboard work area 102 (S64).

In this way, even if the number of pieces of language information in theconversation log area 101 increases as the discussion progresses, theuser can easily review the language information.

Translation of Language in Conversation Log Area

If the information processing system 3 translates the languageinformation displayed in the conversation log area 101, communicationcan be facilitated between users who use different languages. Forexample, when participating in the meeting, the user configures his orher language to be used at the terminal apparatus 1, to cause theterminal apparatus 1 to transmit the configured language to theinformation processing system 3. As described above, the user mayexplicitly select the language. Alternatively, an application running onthe terminal apparatus 1 may read the configured language of theterminal apparatus 1 (for example, the language set at an operatingsystem).

When the user speaks in the meeting, the language information convertedfrom the speech data is shared with the counterpart terminal apparatus9. When the language used on the transmission side is different from thelanguage used on the reception side, the translation unit 35 of theinformation processing system 3 translates the language information intoa language that matches the language used by each user. The translationmay be performed by the counterpart terminal apparatus 9 that receivesthe language information.

FIG. 27 is a sequence diagram illustrating processing of translatinglanguage information, and transmitting the translated languageinformation to the counterpart terminal apparatus 9, performed by theinformation processing system 3, according to the embodiment. In thefollowing description of FIG. 27 , differences from the processing ofFIG. 12 are mainly described.

In FIG. 27 , at steps S121 and S122, the terminal apparatus 1 and thecounterpart terminal apparatus 9 each transmit the used language. AtS123, the translation unit 35 determines whether or not the languageused by the terminal apparatus 1 is different from the language used bythe counterpart terminal apparatus 9. When the used languages aredifferent, the translation unit 35 translates the language informationfrom the used language of the terminal apparatus 1 to the used languageof the counterpart terminal apparatus 9, and vice versa.

In this way, each user participating in the meeting can have thelanguage information in his or her own language.

Playback of Discussions in Meeting

It would be convenient if the user can review the entire meeting at anydesired time, such as at a break in the meeting or after the end of themeeting. When the user plays back the meeting for review, the terminalapparatus 1 displays, in the whiteboard work area 102, the contentcorresponding to the language information in the conversation log area101 that transitions by the scrolling. The scrolling may beautomatically performed or may be performed according to the useroperation. As a result, the user can check a flow of discussions, orunder which situations, each content is arranged in the whiteboard workarea 102, only by scrolling the conversation log area 101 inchronological order from the beginning.

In order to start reviewing, the user presses a review button 107 toexecute a review mode, as illustrated in FIG. 28A. Alternatively, themode may be automatically switched to a review mode when the userparticipates again in the meeting that has ended after the end of themeeting. For example, the user may access a link to the online meeting,after the meeting ends.

FIGS. 28A to 28C are diagrams for explaining processing to review thediscussions (content) in the meeting, when the discussion in the meetingare reproduced according to user operation. FIG. 28A illustrates theinformation display screen 100 before the review button 107 is pressed.

In other words, FIG. 28A illustrates the information display screen 100,which reflects the end of the meeting. Pressing of the review button 107initiates the review mode. The user can scroll the language informationin the conversation log area 101 upward by operating a scroll bar 171.The playback unit 22 specifies, in the content display chorologicaltable, the status of the conversation log area, which is determined byscroll of the user. The playback unit 22 then playbacks the status ofthe whiteboard work area, which corresponds to the status of theconversation log area, in the whiteboard work area. In this example, thereproduction means displaying the content of the status of thewhiteboard work area.

In FIG. 28B, the user has scrolled the language information in theconversation log area 101 upward (in a direction going back in time).Therefore, the content in a state in which the time goes back from thetime when the review button 107 is pressed, is displayed in thewhiteboard work area 102. In FIG. 28C, the user further scrolls thelanguage information in the conversation log area 101 upward (in thedirection going back in time). Accordingly, the content in a status inwhich the time goes back far from the status illustrated in FIG. 28B isdisplayed in the whiteboard work area 102.

FIGS. 29A and 29B are diagrams for explaining processing to review thediscussions (contents) in the meeting when the discussions (contents) inthe meeting are automatically reproduced. When the user presses thereview button 107, the terminal apparatus 1 displays an automaticplayback button 172 on the information display screen 100. Asillustrated in FIG. 29A, the conversation log area 101 displays languageinformation at a predetermined timing, among language informationdisplayed in chronological order. The predetermined timing may be thetiming reflecting the oldest status, or the timing when the meetingstarts for that day. If the predetermined timing reflects a start of themeeting, no language information is displayed. The same applies to thewhiteboard work area 102. In the whiteboard work area 102, even if nolanguage information is displayed in the conversation log area 101,content may be displayed. The user can review changes in theconversation log area 101 and the whiteboard work area 102 inchronological order, from a predetermined timing as illustrated in FIG.29A.

When the automatic playback button 172 is pressed, the playback unit 22moves the scroll bar 171 downward at a constant speed. Accordingly, asillustrated in FIG. 29B, the language information displayed in theconversation log area 101 changes. In operation, the playback unit 22specifies the status of the conversation log area 101 in the content logtable, and playbacks the status of the whiteboard work area 102corresponding to the status of the conversation log area 101 in thewhiteboard work area 102.

In FIGS. 28A to 28C and 29A to 29B, the content in the whiteboard workarea is reproduced so as to correspond to scroll of the scroll bar 171.Alternatively, a slider bar independent of the scroll bar 171 may bedisplayed, which controls the content to be reproduced.

FIG. 30 illustrates an example of a slider bar 174. The slider bar 174is displayed, for example, at a lower portion of the whiteboard workarea 102 in response to pressing of the review button 107. The sliderbar 174 displays various points each indicating a specific timing, suchas a previous start point 175, a today start point 176, and a latestpoint 177. When the user presses the today start point 176, the playbackunit 22 playbacks the first status of the conversation log area 101 andthe first status of the whiteboard work area 102, stored in the contentlog storage unit 1001. When the user presses the latest point 177, theplayback unit 22 playbacks the last status of the conversation log area101 and the last status of the whiteboard work area 102, stored in thecontent log storage unit 1001. When the user presses the previous startpoint 175, the playback unit 22 playbacks the status of the conversationlog area 101 and the status of the whiteboard work area 102, both at thestart of the previous meeting in the meeting management table.

FIG. 31 is a flowchart illustrating processing of playing back thewhiteboard work area, performed by the playback unit 22, when executingthe review mode, according to the embodiment. The processing of FIG. 31starts in response to pressing of the review button 107. The user mayscroll the scroll bar 171 or press the automatic playback button 172.

The playback unit 22 acquires the current status of the conversation logarea 101 (S71).

The playback unit 22 acquires the status of the whiteboard work area 102corresponding to the status of the conversation log area 101 acquired atS71, from the content log storage unit 1001 (S72).

The playback unit 22 displays the content of the status of whiteboardwork area 102, which is acquired, in the whiteboard work area (S73).

During execution of the review mode, the playback unit 22 repeatedlyexecutes steps S71 to S73.

As described above, the visual communication system 5 of the presentembodiment is able to display visual information such as an illustrationin the whiteboard work area 102 based on language information convertedfrom speech. The user can easily visualize the communication during themeeting, as the illustration is automatically displayed. That is, theuser does not have to convert the content of conversation (speechcontent) into an illustration, draw the illustration in the whiteboardwork area 102, or acquire the illustration from a network. Since thewhiteboard work area 102 and the conversation log area 101 are displayedon the same screen, the user is able to instantly know a flow ofcommunication in which each content in the whiteboard work area 102 isproduced.

Other Application Examples

The above-described embodiments are illustrative and do not limit thepresent disclosure. Thus, numerous additional modifications andvariations are possible in light of the above teachings within the scopeof the present disclosure. Any one of the above-described operations maybe performed in various other ways, for example, in an order differentfrom the one described above.

In the above-described embodiment, the information display screen 100 isdisplayed on the display 506, however, the information display screen100 may be displayed by a projector. In such case, the visualcommunication system 5 detects the coordinates of the pen tip forhand-drafting by ultrasonic waves. The pen emits an ultrasonic wave inaddition to the light, and the terminal apparatus 1 calculates adistance based on an arrival time of the sound wave. The terminalapparatus 1 determines the position of the pen based on the directionand the distance, and a projector draws (projects) the trajectory of thepen based on stroke data.

The number of terminal apparatus 1 may be one. In this case, the usercan use the terminal apparatus 1 as a tool for summarizing or recordinghis or her own idea, not during the meeting.

In the block diagram such as FIG. 4 , functional units are divided intoblocks in accordance with main functions of the terminal apparatus 1 andthe information processing system 3, in order to facilitateunderstanding the operation by the terminal apparatus 1 and theinformation processing system 3. No limitation to a scope of the presentdisclosure is intended by how the processes are divided or by the nameof the processes. The processes implemented by the terminal apparatus 1and the information processing system 3 may be divided to a largernumber of processes depending on the contents of processes.Alternatively, one unit of processing may be a group of dividedprocessing.

The apparatuses or devices described in one or more embodiments are justone example of plural computing environments that implement the one ormore embodiments disclosed herein. In some embodiments, the informationprocessing system 3 includes multiple computing devices, such as aserver cluster. The multiple computing devices are configured tocommunicate with one another through any type of communication link,including a network, a shared memory, etc., and perform processesdisclosed herein.

Further, the information processing system 3 can be configured to sharethe processing steps disclosed in the embodiments described above, forexample, the processing steps illustrated in FIG. 12 , in variouscombinations. For example, a process executed by a predetermined unitmay be executed by a plurality of information processing apparatusesincluded in the information processing system 3. The informationprocessing system 3 may be integrated into one server device or may bedivided into a plurality of devices.

Each of the functions of the described embodiments may be implemented byone or more processing circuits or circuitry. Here, the “processingcircuit” in the present specification includes a processor programmed toexecute each function by software such as a processor implemented by anelectronic circuit, and a device such as an application specificintegrated circuit (ASIC), a digital signal processor (DSP), a fieldprogrammable gate array (FPGA), or a conventional circuit moduledesigned to execute each function described above.

The tables according to the above-described embodiments may be generatedby machine learning.

In the present disclosure, the machine learning is defined as atechnology that makes a computer to acquire human-like learning ability.In addition, the machine learning refers to a technology in which acomputer autonomously generates an algorithm required for determinationsuch as data identification from learning data loaded in advance andapplies the generated algorithm to new data to make a prediction. Anysuitable learning method is applied for machine learning, for example,any one of supervised learning, unsupervised learning, semi-supervisedlearning, reinforcement learning, and deep learning, or a combination oftwo or more those learning.

Further, in any one of the above-described embodiments, the languageinformation is displayed in the conversion log area 101, and the contentbased on the language information is displayed in the whiteboard workarea 102. In one or more other examples, the language information andthe content based on the language information may be displayeddifferently, as long as the language information and the content basedon the language information are displayed in different coordinatepositions on the information display screen 100.

Further, the present disclosure describes the case where a plurality ofitems of language information is displayed. However, even when there isonly one language information being displayed in the conversion log area101, the user may select such language information for displayed inwhiteboard work area 102. Similarly, the user may select all items oflanguage information, displayed in the conversion log area 101, fordisplay in the whiteboard work area 102.

Further, as described above, while the speech data is input based onutterances of the users at a plurality of terminal apparatusesparticipating in the conference, there may a case where only one user atthe terminal apparatus may be speaking. In such case, the speech databased on utterances of the user who is speaking is input for display, atleast during a time when the user is speaking.

Aspect 1

According to an exemplary embodiment, a communication system includes aninformation processing system that communicates with a terminalapparatus and another terminal apparatus via a network. Thecommunication system includes: a converter that converts speech databased on speech input to each of the terminal apparatus and the otherterminal apparatus into language information; and a first displaycontrol unit that displays the language information on an informationdisplay screen of the terminal apparatus in a chronological order; asecond display control unit that displays, on the information displayscreen, content based on a user operation on the language informationdisplayed on the information display screen; and a content manager thattransmits the content displayed on the information display screen to theother terminal apparatus.

Aspect 2

The communication system according to Aspect 1 further includes anoperation input unit that receives selection of the language informationdisplayed in a first area of the information display screen. The seconddisplay control unit displays the content based on the languageinformation for which the selection is received at the operation inputunit, in a second area of the information display screen.

Aspect 3

In the communication system according to Aspect 2, the operation inputunit receives an operation that instructs to display, in the secondarea, a part of or the entire language information displayed in thefirst area.

Aspect 4

In the communication system according to Aspect 3, the operation thatinstructs to display includes an operation of dragging and dropping thepart of or the entire language information to a desired location in thesecond area, or an operation of clicking the part of or the entirelanguage information.

Aspect 5

The communication system according to Aspect 2, further includes asearching unit that searches for an image using a character stringincluded in the language information displayed in the first area forwhich the selection is received at the operation input unit. The firstdisplay control unit displays the image searched by the searching unit,in vicinity of the language information for which the selection isreceived at the operation input unit. The operation input unit receivesan operation that instructs to display the displayed image in the secondarea.

Aspect 6

In the communication system according to Aspect 5, the operation thatinstructs to display includes an operation of dragging and dropping theimage that is searched and displayed to a desired location in the secondarea, or an operation of clicking the image that is displayed in thefirst area.

Aspect 7

The communication system according to Aspect 3 further includes asearching unit that searches for an image using a character stringincluded in the language information displayed in the second area. Thesecond display control unit displays the image searched by the searchingunit in vicinity of the language information displayed in the secondarea, and displays the image selected from the displayed image in thesecond area.

Aspect 8

In the communication system according to any one of Aspects 5 to 7, theimages displayed in vicinity of the language information are arranged inan order of appearance of the character strings respectively associatedwith the displayed images.

Aspect 9

In the communication system according to Aspect 3, the second displaycontrol unit decomposes the language information displayed in the secondarea into a plurality of character strings, and arranges the pluralityof character strings as separate contents, in response to a useroperation. In this example, the part of or the entire languageinformation that is selected may be decomposed.

Aspect 10

The communication system according to Aspect 6 further includes ahierarchization unit that hierarchizes the plurality of characterstrings arranged as separate contents, according to a specifiedhierarchical structure. The second display control unit displays theplurality of character strings, while connecting the plurality ofcharacter strings with a line, according to the specified hierarchicalstructure.

Aspect 11

The communication system according to Aspect 6 further includes amachine learning unit that generates a learned model, which converts theplurality of character strings arranged as separate contents accordingto the hierarchical structure.

The second display control unit displays the plurality of the characterstrings in the hierarchical structure, converted by the learned model.

Aspect 12

In the communication system according to Aspect 1, the contents arrangedin the second area of the information display screen are associated withidentification information of the language information, displayed in thefirst area of the information display screen.

When the operation input unit receives a selection of a particularcontent in the second area, the first display control unit highlightsparticular language information in the first area associated withidentification information of the particular content for which theselection is received.

Aspect 13

In the communication system according to Aspect 8, when theidentification information of the language information is not associatedwith the particular content in the second area, for which the selectionis received by the operation input unit,

the first display control unit displays, with emphasis, the languageinformation in the first area having an input time within apredetermined period of time from a time when the selected content isarranged.

Aspect 14

The communication system according to any one of Aspects 1 to 8, furtherincludes a filtering unit that selects the language information to bedisplayed in the first area of the information display screen, such thatthe number of language information displayed in the first area isreduced.

The first display control unit displays only the language informationthat is selected by the filtering unit in the first area.

Aspect 15

In the communication system according to Aspect 14, the filtering unitselects the language information displayed in the first area usinginformation on a speaker of the language information displayed in thefirst area, the number of favorable responses to the languageinformation displayed in the first area, the language information storedby a user as preferable language information, or an input time of thelanguage information to the first area.

Aspect 16

In the communication system according to Aspect 15, the contentsarranged in the second area of the information display screen based onthe language information displayed in the first area are associated withidentification information of the language information.

When the first display control unit displays only the languageinformation selected by the filtering unit in the first area,

the second display control unit arranges only the contents associatedwith the identification information of the language informationdisplayed in the first area in the second area.

Aspect 17

In the communication system according to any one of Aspects 1 to 12, thelanguage information displayed in the first area of the informationdisplay screen are shared between a plurality of terminal apparatusesincluding the terminal apparatus.

The communication system further includes a translation unit thattranslates the language information converted by the converter into alanguage set for the other terminal apparatus, when the language of thelanguage information converted by the converter is different from thelanguage set for the other terminal apparatus.

Aspect 18

The communication system according to any one of Aspects 1 to 13,includes: a log storage processor that stores a status of the first areaof the information display screen, and a status of the second area ofthe information display screen, in association with each other inchorological order; and

a playback unit that playbacks the status of the second area associatedwith the status of the first area.

Aspect 19

In the communication system according to Aspect 18, the playback unitautomatically changes the status of the first area at a constant speed.The playback unit playbacks the status of the second area associatedwith the status of the first area that automatically changes.

Aspect 20

In the communication system according to Aspect 18, the playback unitplaybacks the status of the second area associated with the status ofthe first area that is changed by scroll of the first area by the user.

Aspect 21

According to an exemplary embodiment, a non-transitory recording mediumstoring a plurality of instructions which, when executed by one or moreprocessors at a terminal apparatus that communicates with an informationprocessing system and another terminal apparatus via a network, causethe processors to perform the following method. The method includes:receiving, via the information processing system, language informationconverted from speech data input to the other terminal apparatus andreceived at a voice input unit, and content based on a user operation onthe language information at the other terminal apparatus; displaying, onan information display screen of the terminal apparatus, the languageinformation converted from speech data that is received at a voice inputunit of the terminal apparatus, and the language information of theother terminal apparatus that is received, in a chronological order; anddisplaying the content that is received in the information displayscreen.

Aspect 22

According to an exemplary embodiment, a non-transitory recording mediumstoring a plurality of instructions which, when executed by one or moreprocessors at an information processing system that communicates with aterminal apparatus and another terminal apparatus via a network, causethe processors to perform the following method. The method includes:converting speech data based on speech input to each of the terminalapparatus and the other terminal apparatus into language information;displaying the language information on an information display screen ofthe terminal apparatus in a chronological order; displaying, on theinformation display screen, content based on a user operation on thelanguage information displayed on the information display screen; andtransmitting the content displayed on the information display screen tothe other terminal apparatus.

Aspect 23

According to an exemplary embodiment, a terminal apparatus communicateswith an information processing system and another terminal apparatus viaa network. The terminal apparatus includes a communication unit thatreceives, via the information processing system, language informationconverted from speech data input to the other terminal apparatus andreceived at a voice input unit, and content based on a user operation onthe language information at the other terminal apparatus. The terminalapparatus further includes: a first display control unit that displays,on an information display screen of the terminal apparatus, the languageinformation converted from speech data that is received at a voice inputunit of the terminal apparatus, and the language information of theother terminal apparatus that is received, in a chronological order; anda second display control unit that displays the content that is receivedin the information display screen.

Aspect 24

According to an exemplary embodiment, an information processing systemcommunicates with a terminal apparatus and another terminal apparatusvia a network, includes: a converter that converts speech data based onspeech input to each of the terminal apparatus and the other terminalapparatus into language information; a first display control unit thatdisplays the language information on an information display screen ofthe terminal apparatus in a chronological order; and a content managerthat transmits content displayed on the information display screen andfor which a user operation is received, to the other terminal apparatusfor display.

Aspect 25

According to an exemplary embodiment, an information processing methodperformed by a communication system including an information processingsystem that communicates with a terminal apparatus and another terminalapparatus via a network. The information processing method includes:converting speech data based on speech input to each of the terminalapparatus and the other terminal apparatus into language information;displaying the language information on an information display screen ofthe terminal apparatus in a chronological order; displaying, on theinformation display screen, content based on a user operation on thelanguage information displayed on the information display screen; andtransmitting the content displayed on the information display screen tothe other terminal apparatus.

Aspect 26

According to an exemplary embodiment, a communication system includes:an information processing server including first circuitry; and aterminal apparatus configured to communicate with a counterpart terminalapparatus via the information processing server, the terminal apparatusincluding second circuitry. The first circuitry is configured to:convert speech data into language information, the speech datarepresenting speech input to the terminal apparatus or speech input tothe counterpart terminal apparatus; and transmit the languageinformation each to the terminal apparatus and the counterpart terminalapparatus. The second circuitry is configured to: display the languageinformation that is converted in a first area of an information displayscreen in a chronological order; and display, in a second area of theinformation display screen, content based on a user operation onparticular language information of the language information displayed inthe first area of the information display screen. The first circuitry isconfigured to transmit information on the content displayed on theinformation display screen to a counterpart terminal apparatus, to causethe counterpart terminal apparatus to display the content.

The above-described embodiments are illustrative and do not limit thepresent invention. Thus, numerous additional modifications andvariations are possible in light of the above teachings. For example,elements and/or features of different illustrative embodiments may becombined with each other and/or substituted for each other within thescope of the present invention. Any one of the above-describedoperations may be performed in various other ways, for example, in anorder different from the one described above.

The functionality of the elements disclosed herein may be implementedusing circuitry or processing circuitry which includes general purposeprocessors, special purpose processors, integrated circuits, applicationspecific integrated circuits (ASICs), digital signal processors (DSPs),field programmable gate arrays (FPGAs), conventional circuitry and/orcombinations thereof which are configured or programmed to perform thedisclosed functionality. Processors are considered processing circuitryor circuitry as they include transistors and other circuitry therein. Inthe disclosure, the circuitry, units, or means are hardware that carryout or are programmed to perform the recited functionality. The hardwaremay be any hardware disclosed herein or otherwise known which isprogrammed or configured to carry out the recited functionality. Whenthe hardware is a processor which may be considered a type of circuitry,the circuitry, means, or units are a combination of hardware andsoftware, the software being used to configure the hardware and/orprocessor.

1. A communication system for controlling communication between aterminal apparatus and a counterpart terminal apparatus, comprisingcircuitry configured to: convert speech data into language information,the speech data representing speech input to the terminal apparatus orspeech input to the counterpart terminal apparatus; display the languageinformation that is converted on an information display screen at theterminal apparatus in a chronological order; display, on the informationdisplay screen, content based on a user operation on the languageinformation displayed on the information display screen; and transmitinformation on the content displayed on the information display screento a counterpart terminal apparatus, to cause the counterpart terminalapparatus to display the content.
 2. The communication system of claim1, wherein the information display screen includes a first area and asecond area, and in response to receiving the user operation thatselects language information from among the language informationdisplayed in the first area of the information display screen, thecircuitry displays the content based on the selected languageinformation, in the second area of the information display screen. 3.The communication system of claim 2, wherein the user operation selectsa part of or the entire selected language information displayed in thefirst area of the information display screen, and the circuitrydisplays, as the content, the selected part of or the entire selectedlanguage information, in the second area of the information displayscreen.
 4. The communication system of claim 2, wherein the circuitry isfurther configured to search for an image that matches a characterstring included in the selected language information displayed in thefirst area, display the image that is searched in vicinity of theselected language information, and receive the user operation thatinstructs to display the image that is searched and displayed, as thecontent to be displayed in the second area.
 5. The communication systemof claim 3, wherein the circuitry is further configured to: search forone or more images that match one or more character strings included inthe language information displayed in the second area, display the oneor more images that are searched in vicinity of the language informationdisplayed in the second area, and display an image based a particularimage selected from the one or more images being displayed, in thesecond area.
 6. The communication system of claim 4, wherein, when theimage that is displayed in vicinity of the language information includesa plurality of images respectively associated with a plurality ofcharacter strings included in the language information, the circuitryarranges the plurality of images in an order of arrangement of thecharacter strings in the language information.
 7. The communicationsystem of claim 3, wherein the circuitry is further configured todecompose the selected language information displayed in the second areainto a plurality of character strings, and arrange the plurality ofcharacter strings as contents to be displayed separately from eachother, in response to a user operation.
 8. The communication system ofclaim 7, wherein the circuitry is further configured to display theplurality of character strings in hierarchy, while connecting theplurality of character strings with a line, according to a specifiedhierarchical structure.
 9. The communication system of claim 8, whereinthe circuitry is configured to display the plurality of characterstrings in hierarchy, using a learned model for converting the pluralityof character strings according to the specified hierarchical structure.10. The communication system of claim 1, wherein the content displayedon the information display screen includes a plurality of contents, andin response to receiving a selection of a particular content of theplurality of contents, the circuitry is configured to acquireidentification information that is associated with the particularcontent, and display, with emphasis, language information identifiedwith the identification information that is acquired.
 11. Thecommunication system of claim 10, wherein, in a case where theidentification information that is associated with the particularcontent is not acquired, the circuitry is configured to display, withemphasis, language information having an input time within apredetermined period of time from a time when the particular content isdisplayed.
 12. The communication system of claim 1, wherein thecircuitry is further configured to select the language information to bedisplayed on the information display screen, and display only thelanguage information that is selected.
 13. The communication system ofclaim 12, wherein the circuitry is configured to select the languageinformation to be displayed using at least one of: information onspeaker of the speech data from which the language information isgenerated; a number of favorable responses to the language information;information indicating whether the language information has been storedby a user as preferable language information; or an input time of thelanguage information to the information display screen.
 14. Thecommunication system of claim 13, wherein the content displayed on theinformation display screen includes a plurality of contents, and in acase where the circuitry displays only the language information havingbeen selected, the circuitry is configured to display only one or morecontents each associated with identification information for identifyingthe language information having been selected for display.
 15. Thecommunication system of claim 1, wherein the circuitry is furtherconfigured to translate the language information that is converted intoa language set for the counterpart terminal apparatus, in a case wherethe language converted is different from the language set for thecounterpart terminal apparatus.
 16. The communication system of claim 1,wherein the information display screen includes a first area and asecond area, the circuitry is further configured to: store, in a memory,a status of the first area of the information display screen, and astatus of the second area of the information display screen, inassociation with each other in a chronological order; and playback amoving image based on the status of the second area associated with thefirst area.
 17. The communication system of claim 16, wherein thecircuitry is configured to automatically change the status of the firstarea at a constant speed, such that the status of the second areachanges according to the status of the first area that automaticallychanges.
 18. The communication system of claim 17, wherein the circuitryis configured to change the status of the second area according to thestatus of the first area that changes according to a scroll operation ofthe first area by a user.
 19. An information processing systemcomprising circuitry configured to: convert speech data into languageinformation, the speech data representing speech input to a terminalapparatus or speech input to a counterpart terminal apparatus, theterminal apparatus and the counterpart terminal apparatus beingconfigured to communicate via a network; display the languageinformation that is converted on an information display screen of theterminal apparatus in a chronological order; and in response to a useroperation on the language information displayed on the informationdisplay screen, transmit information on content displayed based on theuser operation to the counterpart terminal apparatus, to cause thecounterpart terminal apparatus to display the content.
 20. Aninformation processing method comprising: converting speech data intolanguage information, the speech data representing speech input to aterminal apparatus or speech input to a counterpart terminal apparatus,the terminal apparatus and the counterpart terminal apparatus beingconfigured to communicate via a network; displaying the languageinformation that is converted on an information display screen of theterminal apparatus in a chronological order; and in response to a useroperation on the language information displayed on the informationdisplay screen, transmitting information on content displayed based onthe user operation to the counterpart terminal apparatus, to cause thecounterpart terminal apparatus to display the content.